Arxiu de la categoria: plone
Thanks to the Plone Foundation that is sponsoring a new dedicated server for our jenkins nodes (the machines that run our test suite on every change),
the Plone community is starting to enjoy faster builds (twice as fast!).
If your pull request or jenkins job runs on Node 5 or Node 6 you will notice it :-)
Please report any misbehavior on jenkins.plone.org github project if you happen to notice something not working as expected.
As the Zope community is getting closer and closer to make a Zope release Python 3 compatible, us, the Plone community have to step up and do the same.
For that, we are working on, guess what, a new Jenkins job that will only run the test suite of all packages that are known to work on Python 3 already.
The initial list isn’t that big, roughly 10 packages so far, but as more and more Zope packages are updated, the more Plone packages that can be made compatible as well.
The upcoming Alpine Sprint will be dedicated towards this: getting a Plone version compatible with the current Zope versions, which will eventually lead to this Zope on python 3 target (aimed to be released by the end of next year).
Once in a while you encounter branches that are have been over-used, i.e. multiple persons added commits there for unrelated issues.
How to get around/solve that? I’m glad you ask :-)
Today I was facing something similar at work, where the messed up branch in question had some commits already merged into master but there were some important other parts to be extracted from it.
So I used three tools:
git log --graph --decorate --pretty=oneline --abbrev-commit --all(with an alias
- a git graphical visualizer (gitg for instance)
- a simple plain text editor (gedit for instance)
git fulllog in a terminal and copy & paste as much information as you need into the text editor.
Fire up the git graphical visualizer and check each relevant commit what’s doing.
Once you know what it is about annotate each commit so you know which commits relate together, i.e. from
ae0ee04 My commit message
ae0ee04 11111 My commit message
This way once everything is annotated is really simple to just grab all the
11111 commits and
git cherry-pick them in a new branch :-)
Pro tip: as you are editing a plain text file you can keep removing lines and adjusting the indentation, suddenly you realize how things keep fitting together!
After a Plone Conference there’s always a lot of energy and willingness to move Plone one step forward.
The testing and CI team in Plone is no different, and for that I created a Doodle to see who is interested and what would be the best time to meet during next week.
If you are interested in the team, or know anyone who would be, please share that link and join the meeting!
2020 is approaching fast, one day at a time, and besides being a nice catchy year, it will be also the year where Python 2.7 will no longer get any security updates.
What that means for Plone? We should hurry up and get a python 3 porting of the whole stack, yes, including Zope, ZTK, ZODB and all the tooling around (zc.buildout, etc etc).
Fortunately quite some tooling and Zope/ZTK/ZODB is already updated and there’s ongoing effort on porting the remaining parts.
The big elephant on the room blocking any porting effort of Plone to python 3 was RestrictedPython, a python distribution that, quoting itself: provides a restricted execution environment for Python.
Note the past on the previous sentence.
Since RestrictedPython is being worked, now it’s high time for the other python distributions from Plone to be also made compatible with Python 3. Stay tuned for the Plone Conference 2016 sprint report!
Copy&pasting&adapting a set of scripts to track the progress of the porting for the Zope foundation github organization, results here, I made the same but tracking what Plone 5.1 (including our testing environment) looks like on Python 3:
During the Plone Conference 2016 there is quite some work put on either reducing the amount of dependencies, or updating our stack to use newer (already python 3 compatible) part of the underlying stack.
The clock is ticking and Plonistas all over the world are working hard on it!
That work is mostly still pending to be merged, because we had a blocker: our current CI infrastructure (namely jenkins master and its nodes) run on a version of Ubuntu1 that does not provide python 3.5 by default.
So it would be a bit irresponsible to merge those changes without a way to ensure that the effort that was put during that sprint is not long forgotten and had to be done again from scratch some months later.
As reports about the newer Ubuntu version regarding python and plone are not that encouraging, and also due to other reasons, I decided to take the longest but long-term best approach: enters gforcada.compile-python!
An ansible role
So I decided to create an ansible role, given that our jenkins CI setup is already using some, and Plone community is also favoring it, to install all the system dependencies to compile Python 2.6, 2.7 and 3.5.
As extras I added:
- install virtualenv on python 2.6 and 2.7 (on python 3.5 is already available)
- install system dependencies for Pillow and lxml
See its README for all the details and more.
With that and vagrant I was able to test that a buildout.coredev (branch 5.1) runs all its tests without a problem :-)
I was tempted to add pypy as well, but I was too lazy/busy for that, if anyone feels like it, pull requests are always welcome!
I hope you find it useful and happy hacking!
- 14.04 for the curious
This week some Plonistas, unfortunately fewer than expected, meet at IN-Berlin to work towards a brighter future for Plone.
Some of us resumed our work already started in Innsbruck sprint early this year while some other topics grew out of discussions and trying things out.
I will never get to write as good as Paul reporting on Barcelona’s sprint, but hey, I will do my best:
In no special order:
The PLIP about assimilating collective.indexing to Plone core finally got all its tests green and is about to be ready for review, work done by Maik Derstappen and Gil Forcada.
Python 3 support was a hot topic during the sprint, so with that in mind Florian Pilz and Michael Howitz created a new tool, zodb.py3migrate, that allows you to convert existing ZODB databases to python 3! Best of all, its great documentation! Be aware that the migration is done in place!
Our Jens2 team was not quadratic this time around, but Jens Vagelpohl continued trying to tame Pluggable Authentication Service (aka PAS).
Did I say that Python 3 was a hot topic? Thomas Lotze with googling support from Gil Forcada took the problem head on and decided that, as Thomas had not that much time (unfortunately he was around only 3 days), work on cracking hard nuts: C extensions. And indeed he did! AccessControl and DocumentTemplate at least compile on python 3.5 (throwing quite some warnings, but hey, have you ever seen Archetypes being installed?). Best of all is that Python 2.6 and 2.7 already have quite some macros for forward compatibility, so the bits that are compatible were pull requested and they are already merged!
Unfortunately that work, C extensions, came to a halt as we hit with RestrictedPython. After long discussions between Thomas, Gil and Maik we decided to make a video conference with the Barcelona sprint, Eric Bréhault, as well as Alexander Loechel, who couldn’t join is in Berlin, but that he already had worked on the topic while in Insbruck early this year (read about his findings). The discussion went really well and Alexander offered himself to continue working on it, stay tuned!
Maik and Stephan meanwhile where not happy with the constellation of form packages that we currently have (z3c.form, plone.autoform, plone.supermodel, plone.z3cform, plone.app.z3cform…) and made yet another package1… just kidding! No, instead they worked on improving the forms related documentation: getting rid of grok2 and making sure that the documentation is consistent3. For that Stephan created the linked above package and Maik was working on moving the remaining useful bits of plone.directives.form to plone.autoform. Stephan started a discussion on the topic on community.plone.org.
Lastly, I put my jenkins/mr.roboto hat on and added some more functionality to mr.roboto: warn which pull request jenkins jobs need to ran to know if a pull request can be safely merged and auto update buildout.coredev’s checkouts.cfg whenever a pull request gets merged. The logic is fairly trivial, but gathering all the pieces together to drive the control is far from it, and testing is yet a complete other matter (thanks mock!!).
All in all we had fun, lots of things got done or discussed and, as with every sprint, everyone is looking for the next one!
- will be soon moved to collective
- and hidden grok dependencies lying in plone.directives.form and plone.directives.dexterity
- at the beginning there are some fields, and two pages later, magically, fields are different!
Since some time ago Jenkins, our continuous integration system, already reports back to pull requests the job build status, just like other popular CI systems, there’s some documentation on how to trigger jenkins builds on your pull requests.
Some of you might have noticed that since last week is adding status updates on pull requests to check if the authors of the pull request have signed the Plone Contributors Agreement.
Since today it is also checking if pull requests are modifying the changelog entry file (namely CHANGES.rst), this way, no change will go unnoticed.
Next step is to warn about which pull requests jobs need to be run for a given pull request. With the current three major releases (4.3, 5.0 and 5.1) being tested is quite a challenge to know which major versions a pull request should be tested against.
As usual, please report any problem on jenkins.plone.org issue tracker.
The topic this time is to continue the long endeavor to cleanup and slim down our stack, Zope included.
The main communication channels with the sprints will be the usual #sprint IRC channel (on freenode) and we will try to maintain a hangout channel open as well (we will provide the link later on).
On social networks you can try to keep an eye on #berlinsprint hashtag.
You can find all information in the link above.
There are still some spots left, the venue is big enough to accommodate around 15 to 20 persons, so if you want to join the sprint, now is the time to sign up and register. As it’s an strategic sprint the Plone Foundation could help with the travel costs if that’s an issue for you to attend, contact them!
And best of all: spring has already arrived in Berlin, so we can expect good weather1 and a full city to enjoy!
See you all in Berlin!
- hopefully not famous last words
Although for me it looks clear and straightforward, for some it may not be the case, so I decided to add a brief document explaining it. It should show up in Read The Docs here: http://jenkinsploneorg.readthedocs.org/en/latest/run-pull-request-jobs.html
If Read The Docs still hasn’t catch up the source in GitHub is easily readable as well: https://github.com/plone/jenkins.plone.org/blob/master/docs/source/run-pull-request-jobs.rst
Bonus point: I made my first screencast demoing it! Watch it in all its glory.
0 to 1k
Yesterday marked the day that Jenkins Job “Pull Request 5.0” hit the 1000 job (right now running the 1016!).
It’s been a long journey to get it to its current status1 but IMHO since the introduction of it our three main Jenkins Jobs for both 4.3 and 5.0 have been far more stable.
Thanks to everyone that reported feedback and is using it!
100k to 0?
Jenkins is not only about tests, code analysis and all other kind of jobs are running on http://jenkins.plone.org. Two of the latest additions are:
- per package code analysis jobs that report back to github: packages can opt-in2 and are checked with flake8 and other tools3
- a global code analysis job: exactly the same as the per package jobs mentioned on the previous point, but running code analysis on ALL packages
The 100k is the flake8 error count for the global job, will we be able to bring that down to zero? :-)
Side note: Zope2 and CMFCore are the two (by far) with more code analysis errors, some other packages are probably going to be deprecated so there is no need to clean up them.
Best of both things?
Anyone (you!?) can grab a package clean it up, and run a pull request job to ensure nothing is broken after the clean up and happily merge it.
To add sugar on top, the packages that are already monitored for it, point you to that same guide, see it in action.
Who cares what a style guide says regarding indentation, dependencies, string quoting, docstrings, handling i18n, etc etc if then one can freely commit changes that go against the said style guide?
Talk is cheap
I’ve been putting as much effort and free time as I could to make that happen.
plone.recipe.codeanalysis has been improved1, flake8 plugins have been written, and finally during this last Alpine City Strategic Sprint 2016 I could implement the missing piece: report back to the users (see an example)!
The script is fairly simple:
#!/usr/bin/env bash wget https://raw.githubusercontent.com/plone/buildout.coredev/5.0/bootstrap.py -O bootstrap.py wget https://raw.githubusercontent.com/plone/buildout.coredev/5.0/experimental/qa.cfg -O qa.cfg wget https://raw.githubusercontent.com/plone/plone.recipe.codeanalysis/master/.isort.cfg -O .isort.cfg python bootstrap.py -c qa.cfg bin/buildout -c qa.cfg bin/code-analysis
Run the above on any Plone distribution and see the output collected on parts/code-analysis/flake8.log
Currently only a few packages are checked, but cleanup tasks can start on any Plone package as of now. Be sure to ask for a job that will check that the package remains clean as soon as your cleanup changes are merged!
- and by no means only by me
This past five days I had the pleasure the be part on the last Plone sprint: Alpine City Strategic Sprint 2016, what a blast!
In the following days, the amazing never running out of steam, Jens will report back to the community. A partial/preview report already exists for the impatient ones (disclaimer: I made it, so there’s probably quite a few things missing).
Below I will make a summary of what I achieved thanks to being part of the sprint.
- bring back node4 on Jenkins: meaning that we can run more jobs at the same time, really handy for a sprint or at release time/high activity!
- quickly update (totally untested, so dragons ahead) ZODB, Zope, ZTK and CMF jenkins jobs, now they are bundled together in a tab. Pointers on how to make them reliable/improve them highly appreciated, submit tickets for it so we can keep track of them
- create jenkins jobs for PLIPs being worked on during the sprint. In retrospect they took me less than 5 minutes of work1 and they proved to be extremely helpful. Pro tip: whenever working on a new PLIP ask for a jenkins job!
- create more jenkins jobs for checking, and reporting back, code analysis on distributions (more about it on a follow-up post)
Fortunately all the above took less than one or two hours, my main task during the sprint was, together with Maik, bring collective.indexing in Plone core without having to add yet another package2. Please read the PLIP description, linked above, to know the scope of it.
As of now we are down to one single test failure! We are highly appreciated on creative ways to solve that last one…
Not only work
A sprint is way more than just sitting in front of a laptop and coding!
Lots of interesting discussions where held, we had lots of fun, we did some sightseeing and enjoyed being together.
And after that I can only say that Plone’s future is brighter than ever has been!
As a personal pet peeve, we finally agreed (?) on how to sort imports! :-)
Last words go to Jens and Christine for taking so much care on preparing everything and being so open and helpful at any time, thanks!
- thanks to jenkins-job-builder
- which our release manager will probably welcome for a change
I’m glad you asked, because there’s plenty of them, for example these ones:
There a step by step tutorial on the first message and plenty of packages to choose from.
World Plone Office Day is 3 weeks away only so, whether you are hosting one office day or you are thinking of to attend it, be sure to get a list of easy tasks to do for newcomers.
It probably happened to you, dear reader, every now an then: the new feature you are working on has changes across more than one git repository, so how can this be tested to make sure nothing is broken before merging those nth separate pull requests?
Let us know if it does not work as expected!
Now with Mockup he have the other side of the coin as well: we can not only check how the code integrates within Plone but we can also check its logic with fast unittests.
There’s already Travis integration for Mockup and Mockup-core (a building block for Mockup), but as Plone is using Jenkins it makes a lot of sense to run them there as well.
So, how that’s done? I’m glad you ask, exactly like this:
- Add Nodejs on Jenkins nodes (plone.jenkins_node)
- Configure Jenkins so that tests will find where Nodejs is (plone.jenkins_server)
- Add a make target for the new Grunt configuration (mockup)
- Create a Jenkins job that runs it (jenkins.plone.org)
All the code is there, the last three pull requests are pending to be merged.
P.D: remember that tomorrow is WPOD!
This Friday Mai 29th we are celebrating the 2nd edition of WPOD!
Look around your city, meet with friends, make new ones and contribute to Plone!
As usual IRC will be full of contributors willing to help smooth your path into Plone, so don’t be shy and ask ;)
See you there/in IRC!
Did you ever got tired waiting a repository (managed by mr.developer) to be cloned?
Wait no more! mr.developer 1.32 comes with two new options to make your life easier:
- a global git-clone-depth option that allows to fetch only the amount of history specified on this option (i.e. git-clone-depth = 1 for only one single commit)
- a per-source depth option that allows to specify the same as git-clone-depth but only for that specific source
What’s the benefit of this? See it for yourself:
git clone https://github.com/plone/buildout.coredev.git test-full du -sh test-full 36M test-full git clone https://github.com/plone/buildout.coredev.git --depth=1 test-shallow du -sh test-shallow 1,5M test-shallow
That’s a 24x saving.
So think about your CI environments where they are downloading over and over the whole repository information while you only care about the very last changes.
Think about some remote environments (mobile connection while you are on a train?)
Update: it seems that is completely broken, sorry for that, working on a fix.
Update 2: mr.developer 1.33 released, thanks @fschulze for the new release!
Maybe everyone is already aware of them, but just in case…
zc.buildout is THE building block that assembles Plone together.
It’s been around for quite a while (+10 years) and it has plenty of features.
Two of them which I’m enjoying a lot lately are:
./bin/buildout install code-analysis
install allows you to override which parts are going to be installed and thus it allows, like in the example above, reduce the amount of packages to fetch and things to build. Which for something like jenkins.plone.org can make quite a lot of sense to use it more and more.
annotate outputs the complete configuration that buildout will use as if it was everything in a single file. This is great for debugging “why my configuration is overriden or not being used at all” kind of errors.
Last Friday the first WPOD happened around the globe.
Here is what was done on the Berlin gathering:
- Work on plone.app.querystring continued (specially on pull request 31)
- Some work was done on fixing and improving the coredev docs
- A brown bag release was spotted on the wild
- jenkins.plone.org nodes use the newer ansible playbook plone.jenkins_node
- a pull request for it has been created as well
- lots of fun and knowledge sharing!
Hopefully other participants around the globe will share their achievements as well!
Are you already planning the next WPOD? Mark it on your calendar May 29th.
Thus making it feasible to spot where the problem was introduced, find the changes that were made and report back to the developer that made the changes to warn him/her about it.
A more elaborate step by step description is on our CI rules, be sure to read them!
At the same time though, we use GitHub pull requests to make code reviews easy and have a way to provide feedback and context to let everyone give their opinion/comment on changes that can be non-trivial.
Sadly, pull requests and multi-repository changes can not be tested directly with Jenkins, yes, there is a plugin for that, but our CI setup is a bit (note the emphasis) more complex than that…
Fortunately we came up with two solutions (it’s Plone at the end, we can not have only one solution :D)
Single pull requests
If you have a pull request on a core package that you want to test follow these steps:
- Get the pull request URL
- Go to http://jenkins.plone.org and login with your GitHub user
- Go to pull-request job: http://jenkins.plone.org/job/pull-request (you can see it always at the front page of jenkins.plone.org)
- Click on the Build with Parameters link on the left column
- Paste the pull request URL from #1 step
- Click on Build
Once it runs you will get an email with the result of the build. If everything is green you can add a comment on the pull request to let everyone know that tests pass.
Note: it’s your responsibility to run that job with your pull request and that changes made on other packages after tests started running can still make your pull request fail later on, so even if the pull-request job is green, be sure to keep an eye on the main jenkins jobs as soon as you merge your pull request.
Example: Remove Products.CMFDefault from Products.CMFPlone (by @tomgross)
Pull request: https://github.com/plone/Products.CMFPlone/pull/438
Jenkins job: http://jenkins.plone.org/job/pull-request/80
When the changes, like massive renamings for example, are spread over more than one repository the approach taken before doesn’t work, as the pull-request Jenkins job is only able to change one single repository.
But we, the CI/testing team, have another ace on our sleeves: create a buildout configuration in the plips folder on buildout.coredev (branch 5.0) that lists all your repositories and which branch should be used, see some examples.
Once you have that cfg file, you can politely ask the CI team to create a Jenkins job for you. They will make
a lot of clever things to make that work on jenkins (3 lines change plus following some instructions) and sooner or later a new Jenkins job will show up on the PLIPs tab on jenkins.plone.org.
Rinse and repeat!
Extra bonus and caveats
All Jenkins jobs, be it the pull-request, PLIPs and of course the core jobs, are configured to send an email to the one that triggered the job, so don’t worry about how long do they take to run, once they are finished you will get notified.
The caveat is that the above is only valid for changes targeting Plone 5. We didn’t put the extra effort to make it sure it also works for pull requests (either single or multi-repository) aimed at Plone 4.3. It’s quite trivial to add it for multi-repositories, a bit more work to make it run on single pull requests, still feasible to do if there’s enough people asking for it.
Hopefully the amount of pull requests for Plone 4.3 will decrease more and more as Plone 5 is getting closer and closer one pull request at a time :)
Now there’s no excuse on pushing changes to master without having tested them before on jenkins.plone.org!
Proposals on improvements and suggestions are always welcome on the issue tracker for jenkins.plone.org GitHub repository. Help on handling all those issues are, of course, also welcomed!
WPOD: World Plone Office Day
During this year’s PLOG I presented the simple idea behind WPOD:
- every last Friday of the month
- meet somewhere (remotely is fine as well, see below)
- hack on Plone instead of your regular work
- Rinse and repeat
That’s it, as simple and as easy as it can be.
Mark on your calendars every last Friday of the month, you have an appointment with the Plone community to bring Plone one step further ahead!
WPOD in Berlin
Preparations are being made for the first ever WPOD in Berlin that my company will gladly host. If you happen to be around Berlin, please contact me telling that you are coming!
The location is Hegelplatz 1, 10117 Berlin.
You are welcome during all day long. Plonistas are expected to come during the morning, enjoy some lunch together, and hack away until late afternoon.
WPOD around the world
If you happen to not be in Berlin, fear not, an irc channel will be available (#sprint on irc.freenode.net) so you can feel the same experience as in any other city hosting a WPOD.
Credit where credit’s due
That’s not an original idea of mine, nor is something that I thought of myself alone, Philip Bauer already tried to present the very same idea on last year’s Plone Conference in Bristol.
Later on, during the Bycicle sprint in Berlin, Stefania and I discussed about it and defined the format as it will start with.
Thanks to them for their bright minds and clever ideas!
Within 10 days the first WPOD will happen, Plonistas will hack/plan/design away and Plone will get better and better.
I hope that other cities and individuals alike will start participating on WPOD, the more we are the bolder the change we will make.
There are some plans to put all the relevant information regarding WPOD on plone.org, either on the current website, or even better on the newer plone.org that is on the making (watch here for tickets ready to be fixed by any of you!).
Update: a meetup has been created, please RSVP there.
the Oscar Gil goes to…
Després de parlar-ne molt amb la Sílvia, amb vol a Berlín inclòs per fer una entrevista en persona, vam decidir que era una bona oportunitat (si m’hi volien clar).
Berlín és una ciutat que sempre m’ha agradat, ja hi he estat tres o quatre vegades, hi coneixem gent, etc etc. A més, per la Sílvia podrà ser una molt bona oportunitat per aprendre un nou idioma. També hi tenim amics i evidentment i com a fet primordial, la feina, no només em sembla bona, sinó també una oportunitat per millorar els meus coneixements de Plone/Zope/Python i tot el que hi corre entre mig.
Començo el dia 7 de novembre, d’aquí quatre dies comptats!!!, així que si heu de passar per Berlín d’aquí unes setmanes, mesos o anys, recordeu que hi haurà una parella, més, de catalans vivint-hi que us poden acollir, fer-vos de guia o portar-vos a prendre alguna cosa tot parlant de la vida i del que sigui :D
Menció especial es mereix el projecte guifi.net i la meva relació amb ell. Des de que vam iniciar el projecte ja farà uns quants anys, encara no he tingut mai connexió de guifi a casa: només començar el projecte vaig anar a viure a Girona per estudiar a la universitat, quan la xarxa s’estava acostant a Girona, i després de fer-hi alguna xerrada per la universitat i parlar-ne amb diverses persones i col·lectius, vam venir a viure a Molins de Rei amb la Sílvia. De nou, aquests dies que estem començant a posar antenes per Molins, agafem i marxem a l’altra punta d’Europa :)
Per altra banda, potser vol dir que d’aquí un temps la xarxa ja ha crescut tant que arriba a Berlín… Qui sap! Però sembla que no ens hi podrem estar molt de temps tampoc per Berlín!
Demà al migdia marxo cap a la Plone Conference!
La Plone Conference és la trobada anual de desenvolupadors, usuaris i actors al voltant del sistema de gestió de continguts Plone.
Aquest any, després d’un any de treballar pràcticament cada dia sobre el Plone crec que en podré treure molt de profit, així ho espero de fet
L’únic punt negatiu serà, com de costum, el fantàstic temps que ens espera (gariebé pluja cada dia!!).
Si algú hi va que em digui alguna cosa!
A diferència del CSS però, els sistemes de plantilles sí que tenen part de programació, iterar, condicionals i coses per l’estil.
Si treballeu amb Plone ja haureu descobert que utilitzeu TAL, un sistema de plantilles que utilitza una sintaxi molt semblant a l’HTML de manera que el propi codi de TAL queda integrat en tot el que és l’HTML que es vol mostrar.
Avui he trobat una referència del llenguatge i per això he fet l’entrada, per recordar-me que existeix