<?xml version="1.0" encoding="utf-8" ?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:syn="http://purl.org/rss/1.0/modules/syndication/" xmlns="http://purl.org/rss/1.0/">




    



<channel rdf:about="http://davisagli.com/front-page/RSS">
  <title>David Glick – Plone developer</title>
  <link>http://davisagli.com</link>

  <description>
    
      
    
  </description>

  

  
            <syn:updatePeriod>daily</syn:updatePeriod>
            <syn:updateFrequency>1</syn:updateFrequency>
            <syn:updateBase>2010-04-06T06:19:44Z</syn:updateBase>
        

  <image rdf:resource="http://davisagli.com/logo.png"/>

  <items>
    <rdf:Seq>
      
        <rdf:li rdf:resource="http://davisagli.com/blog/ten-lesser-known-improvements-in-plone-4"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/plone.app.themeeditor-1.0a1-...-and-a-roundup-of-other-recent-releases"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/in-browser-integration-testing-with-windmill"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/review-of-plone-3-products-development-cookbook"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/introducing-the-plone-resource-customizer"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/be-vigilant-know-your-buildout-threat-level"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/report-out-from-the-seattle-getpaid-recurring-payment-mini-sprint"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/notes-on-migrating-this-blog-from-wordpress-to-plone"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/tahoe-sprint-off-and-running"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/reflections-on-pycon-2010"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/reflections-on-building-a-member-directory-using"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/using-haproxy-with-zope-via-buildout"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/on-zope-multiple-cores-and-the-gil"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/come-improve-dexterity-at-the-tahoe-snow-sprint"/>
      
      
        <rdf:li rdf:resource="http://davisagli.com/blog/mr-igor"/>
      
    </rdf:Seq>
  </items>

</channel>


  <item rdf:about="http://davisagli.com/blog/ten-lesser-known-improvements-in-plone-4">
    <title>Ten lesser-known improvements in Plone 4</title>
    <link>http://davisagli.com/blog/ten-lesser-known-improvements-in-plone-4</link>
    <description>The just-released Plone 4 includes rather a lot of big enhancements. Here are some of my favorite little ones.</description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>As you've probably heard by now, <a class="external-link" href="http://plone.org/4">Plone 4.0 final</a> was released this week! (The cynical among you are probably muttering that we <a class="external-link" href="http://rakudo.org/announce/rakudo-star/2010.08">couldn't even beat Perl 6</a>, but hey, it's here now :) ). Working on Plone 4 has been a huge part of my life for the past year, but I did far from all of the work. Thanks to every one of you who contributed; I couldn't be prouder of what we've created together!</p>
<p>See the link above for info on major features in the release, but there are a lot more little things included too. Without further ado, here are ten of my favorite Plone 4 improvements that nobody's talking about...</p>
<h3>General / site admin features</h3>
<p>1. <b>New editor features.</b> Check the control panel for TinyMCE (our new WYSIWYG editor) to find some useful buttons that aren't enabled by default:</p>
<ul>
<li>Paste as Plain Text (preserves newlines)</li>
<li>Paste from Word (strips junk markup)</li>
<li>Insert/edit Media</li>
</ul>
<p>(among others)</p>
<p>2. <b>Better date/time handling.</b> The display of event times in listings is now smarter. If the event begins and ends on different days, the time is not shown. If the start and end times are the same, it is not shown twice.</p>
<p>In addition, dates and times are now stored with a timezone, with proper consideration of daylight savings time.</p>
<p>3. <b>Inline error tracebacks.</b> If you're logged in as a Manager and encounter an error, the traceback is shown immediately, rather than requiring you to click through to the error_log in the ZMI.</p>
<p>4. <b>Permission auditing.</b> Thanks to a new feature in Zope 2.12, it's now easy to double-check what permissions a particular user will have in a particular context, via this button on the Security tab in the ZMI:</p>
<p style="text-align: center; "><img src="http://davisagli.com/images/Screenshot20100902at11.56.40PM.png" alt="Permission auditing" class="image-inline" /></p>
<h3 style="text-align: left; ">For integrators</h3>
<p style="text-align: left; ">5. <b>Better control over portlet columns</b>. In Plone 3, it was possible to make a template prevent rendering of the portlet columns by filling METAL slots. In Plone 4, this can be done by setting the <i>disable_plone.leftcolumn</i> and <i>disable_plone.rightcolumn</i> flags on the request before calling main_template. For example:</p>
<pre class="xml">&lt;tal:block tal:define="foo python:request.set('disable_plone.leftcolumn', 1)"/&gt;<br />&lt;html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"<br /> xmlns:tal="http://xml.zope.org/namespaces/tal"<br /> xmlns:metal="http://xml.zope.org/namespaces/metal"<br /> xmlns:i18n="http://xml.zope.org/namespaces/i18n"<br /> lang="en"<br /> metal:use-macro="context/main_template/macros/master"<br /> i18n:domain="plone"&gt;<br /><br />(snip)<br />&lt;/html&gt;<br /></pre>
<p>(This might also be done in Python in the __init__ method of a browser view.) Handily, it's actually a 3-way flag: set it to True to force hiding the colum, False to force displaying it even if there are no portlets assigned, and None to use the default logic based on whether there are portlets assigned.</p>
<p>6. <b>"All content" listing view and content-core macro.</b> There is a new folder listing display option called "All content", which renders the full content of each item. (See it in action on the homepage of this website.)</p>
<p>This listing takes advantage of the new <i>content-core</i> slot which is defined in main_template following the title and description and between the various above/below viewlet managers, and filled by each content type's default view. This gives each content type control over how it renders in the "all content" listing. (The "all content" listing also takes advantage of some new slots/macros in the folder_listing template to reuse its logic for batching and fetching item properties; this may be a useful model for other custom folder listings.)</p>
<p>It also makes it simpler to write a view template for a content type, since one no longer needs to include the title, description, and common viewlet managers as boilerplate. The simplest view template is now (this is a copy of Plone 4's document_view):</p>
<pre>&lt;html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"<br /> xmlns:tal="http://xml.zope.org/namespaces/tal"<br /> xmlns:metal="http://xml.zope.org/namespaces/metal"<br /> xmlns:i18n="http://xml.zope.org/namespaces/i18n"<br /> lang="en" metal:use-macro="context/main_template/macros/master"<br /> i18n:domain="plone"&gt;<br /> &lt;body&gt;<br />  &lt;metal:content-core fill-slot="content-core"&gt;<br />   &lt;metal:content-core define-macro="content-core"&gt;<br />    &lt;metal:field use-macro="python:context.widget('text', mode='view')"&gt;<br />     Body text<br />    &lt;/metal:field&gt;<br />   &lt;/metal:content-core&gt;<br />  &lt;/metal:content-core&gt;<br /> &lt;/body&gt;<br />&lt;/html&gt;</pre>
<p>For backwards compatibility, templates may still continue to fill the <i>main</i> slot instead of <i>content-core</i>, and provide their own title, description, and viewlet managers.</p>
<p>7. <b>Flexible image scaling.</b> There is now an <i>Image Handling</i> control panel which allows configuring the standard image scale sizes used by Plone.</p>
<p>In addition, since image scales are now generated on demand, it's quite easy for a template to request an image scale of a non-standard size, using this idiom:</p>
<pre>&lt;img tal:define="scale context/@@images"<br />     tal:replace="structure python: scale.scale('image', width=42, height=42).tag()" /&gt;</pre>
<p>Including a standard image scale is simpler too:</p>
<pre>&lt;img tal:replace="structure context/@@images/image/mini" /&gt;<br /></pre>
<p>See <a class="external-link" href="http://pypi.python.org/pypi/plone.app.imaging">plone.app.imaging</a> for additional options.</p>
<h3>For developers</h3>
<p>8. <b>Automatic registering of Zope 2 permissions.</b> In Zope 2.10, registering a new permission took two steps (using the &lt;permission/&gt; ZCML directive to register a Zope 3 permission utility, and registering it as a Zope 2 permission, usually by calling Products.CMFCore.permissions.setDefaultRoles during product initialization).</p>
<p>In Zope 2.12, only the first step is required.  The equivalent of setDefaultRoles is automatically called after processing the <i>permission</i> directive, and sets the default roles to ('Manager',).</p>
<p>If you want different default roles, you must still call setDefaultRoles yourself, prior to loading of ZCML (e.g., at import time). This will improve in Plone 4.1 / Zope 2.13, where the <i>permission</i> directive gets an optional <i>role</i> subdirective to specify the desired default roles in ZCML.</p>
<p>9. <b>Fixed ordering of CMF/Archetypes interaction during object creation.</b> In Plone 3, when you created a new content item, CMF called the factory method of an Archetypes content type, then called its ObjectCreated event, <i>then</i> updated the item's <i>portal_type</i> attribute. This had the unfortunate effect that if you were trying to create a schema extender that used the portal_type to determine whether the extender should be applied (perhaps based on some configurable property), and your site used content types that shared the class of other content types, you were out of luck.</p>
<p>This is now fixed so that ObjectCreated is not fired until the portal_type is set. I think we also managed to avoid calling some event twice during object creation that was causing duplicate catalog indexing, but I could be misremembering that.</p>
<p>10. <b>Folder ordering adapters.</b> The order of items in a folder is now looked up via a named adapter of the folder to plone.folder.interfaces.IOrdering—and it is looked up dynamically when ordering catalog results by getObjPositionInParent, rather than consulting an index. This means that it should now be simpler to customize how a particular folder is ordered (e.g. to automatically sort alphabetically or by date), by writing a new adapter that provides IOrdering and setting a folder to use it. To set the name of the IOrdering adapter that is used for a particular folder, use its setOrdering method.</p>
<p>Out of the box, most folders use the DefaultOrdering adapter, which allows user-configurable order. This should be appropriate in many cases. Folders that were migrated from Plone 3's Large Plone Folders use the "unordered" ordering (but this shouldn't be needed for new folders that will store many items, as the default ordering is reasonably performant). plone.folder also has a "partial" ordering which allows specifying the order for some but not all subitems, but this is not directly used or supported in the Plone UI at this time.</p>
<hr />
<p> </p>
<p>On to 4.1 and beyond!</p>]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2010-09-03T16:50:19Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/plone.app.themeeditor-1.0a1-...-and-a-roundup-of-other-recent-releases">
    <title>plone.app.themeeditor 1.0a1 ... and a roundup of other recent releases</title>
    <link>http://davisagli.com/blog/plone.app.themeeditor-1.0a1-...-and-a-roundup-of-other-recent-releases</link>
    <description>The Groundwire team and I have been busy bringing you updates to your favorite Plone add-ons.</description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>Sometimes it's easy to get carried away fixing bugs and making new releases without remembering to announce them.  So here's a blog post to catch up on what the <a class="external-link" href="http://groundwire.org">Groundwire</a> web team and I have been up to in Plone add-on land the past few months.<br /><br />Most noteworthily, David Bain has been working on plone.app.themeeditor, an improved UI for customizing Plone theme resources, as his Google Summer of Code (GSOC) project. This week we were finally ready to release the <a class="external-link" href="http://pypi.python.org/pypi/plone.app.themeeditor/1.0a1">first alpha</a>. Please try it out and give us feedback so that we can continue to make it better!  (Note the special installation instructions if you're on Plone 3.)  GSOC is almost over, but David is still at work trying to add support for automatically exporting customized items to a filesystem theme product.  If you missed the blog post where I introduced the theme editor, you can go <a href="http://davisagli.com/blog/introducing-the-plone-resource-customizer" class="internal-link">watch the demo screencast</a>.<br /><br />And a bunch more...<br /><br /><a class="external-link" href="http://plone.org/products/pfg.donationform">pfg.donationform</a> 1.0 - This product simplifies the process of setting up a PloneFormGen-based donation form that processes payments via PloneGetPaid. It provides a custom add form that creates a PloneFormGen form with fields for contact and billing info, and an adapter to fill a cart and initiate checkout. It includes a custom "donation field" and widget for use with PloneFormGen, which allows choosing from a list of predefined named donation levels or choosing an arbitrary amount to donate. It also supports an option to make a recurring donation if you're using a GetPaid payment processor that supports that (currently the Authorize.net and PayPal processors).<br /><br /><a class="external-link" href="http://plone.org/products/pressroom">Products.PressRoom</a> 3.8 - We've been fixing various issues with Plone 4 compatibility, and Matt Yoder just made a change so that the various listings (press releases, press clippings, etc.) are now based on Collections, for improved customizability.<br /><br /><a class="external-link" href="http://plone.org/products/salesforcebaseconnector">Products.salesforcebaseconnector</a> 1.3 - Query results can now be accessed from RestrictedPython in templates, and there is a new validateCredentials method to aid in remotely monitoring to confirm that the Salesforce connection is working.<br /><br /><a class="external-link" href="http://plone.org/products/salesforcepfgadapter">Products.salesforcepfgadapter</a> 1.6.2 - The new release includes a bugfix to support the case when you're using the adapter in update mode and the user is trying to clear a non-required field. (thanks to Matt Yoder)<br /><br /><a class="external-link" href="http://plone.org/products/megaphone">collective.megaphone</a> 1.4 - This new release of Groundwire's online advocacy campaign tool contains some fixes for Plone 4 compatibility, as well as a rewrite of the drag-and-drop UI for reordering fields to avoid a dependency on collective.jqueryui.  Look for a bigger Megaphone release including a new Petition feature soon.<br /><br /><a class="external-link" href="http://pypi.python.org/pypi/collective.salesforce.authplugin">collective.salesforce.authplugin</a> 1.4 - Contains a couple bugfixes, plus some improvements to make it easier to use multiple auth plugins in the same Plone site (i.e. if you need to authenticate against multiple types of Salesforce objects).<br /><br /><a class="external-link" href="http://plone.org/products/simplesocial">collective.simplesocial</a> 1.3 - Adds a couple features including a collective.googleanalytics plugin to track clickthroughs when prompting people to post to their Facebook feed, and an option to only show the latter after an edit. (thanks to Matt Yoder)<br /><br /><a class="external-link" href="http://plone.org/products/collective.z3cform.wizard">collective.z3cform.wizard</a> 1.3.2 - Avoids some spurious inline editing errors.<br /><br /><a class="external-link" href="http://plone.org/products/plone.app.jquerytools">plone.app.jquerytools</a> 1.1.2 - This is mostly Steve McMahon's brainchild, but in the most recent release I added a stylesheet for Plone 3 based on the overlay styles from Plone 4's Sunburst theme, so that add-ons using jquerytools don't have to provide styles for the popup overlays.<br /><br /><a class="external-link" href="http://pypi.python.org/pypi/Products.PloneFormGen">Products.PloneFormGen</a> 1.6.0b4 - We shifted to a different approach for getting keys for storing items in the saved data adapter, which should help avoid conflicts in heavy-write scenarios.<br /><br />Finally, as I write this, Martin Aspeli is making a new beta release of the <a class="external-link" href="http://plone.org/products/dexterity">Dexterity</a> content type system, incorporating numerous bugfixes.<br /><br /></p>]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2010-08-05T05:55:00Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/in-browser-integration-testing-with-windmill">
    <title>In-browser integration testing with Windmill</title>
    <link>http://davisagli.com/blog/in-browser-integration-testing-with-windmill</link>
    <description>Windmill makes it easy to run automated tests of a Plone project in a real browser.</description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>I really like writing integration tests for web projects using the <a class="external-link" href="http://pypi.python.org/pypi/zope.testbrowser">Zope testbrowser</a>, which is a convenience API around the mechanize library. But its Achilles heel has always of course been that it only operates on the HTML response, so can't test interactive functionality built with Javascript and AJAX. So I've wanted to try one of the options for running tests in a real browser for a while. Actually it's a testament to the utility of the Zope testbrowser (or my own laziness?) that I made it this long. But the <a class="internal-link" href="introducing-the-plone-resource-customizer">Plone resource customizer</a> uses a lot of AJAX, so it was time.<br /><br />So first I tried <a class="external-link" href="http://seleniumhq.org">Selenium</a>. I'd heard about Selenium from a number of people, and it's cool. There is a Firefox plugin that lets you record actions and assertions and play them back. It also lets you export these tests to Python to be run through the selenium Python bindings in conjunction with Selenium RC. There's even a Selenium Grid for running tests in parallel on multiple machines.<br /><br />Unfortunately, while I found <a class="external-link" href="http://pypi.python.org/pypi/collective.ploneseltest">collective.ploneseltest</a> which looked like just what I needed—it provides a base Selenium test case for use with the Zope testrunner—I had trouble getting it to actually work. Using version 1.0.3 of Selenium RC and of the Python bindings, Selenium RC was sending an extra HEAD request before each GET request, which was getting interpreted incorrectly by the ZPublisher during traversal in Zope.  A query to Twitter yielded the information (from the <a class="external-link" href="http://twitter.com/hugs">creator of selenium</a> himself) that new Python bindings for selenium 2.0 are available as of last week, but I found they don't work (at least not yet) with Python 2.4. And anyway, by that point Martin Aspeli had also replied to my query and suggested trying Windmill instead.<br /><br /><a class="external-link" href="http://getwindmill.com">Windmill</a> is another web testing framework that actually seems to have quite similar functionality to Selenium, at least from a cursory examination. It also has an in-browser controller for recording, and can export tests to Python code. There is a Zope testrunner integration for Windmill too, in the <a class="external-link" href="http://pypi.python.org/pypi/niteoweb.windmill">niteoweb.windmill</a> package.</p>
<p>And Windmill was quite a bit easier to get working. I just added the following to my package's setup.py to define a new installation "extra":</p>
<pre>extras_require = {<br />    'test': ['niteoweb.windmill',],<br />},</pre>
<p>And then modified my buildout's test runner to include that extra:</p>
<pre>[test]<br />recipe = zc.recipe.testrunner<br />eggs =<br />    ${instance:eggs}<br />    plone.app.skineditor [test]<br />defaults = ['--exit-with-status', '--auto-color', '--auto-progress']</pre>
<p>I added a new test module called test_integration.py and made it use the base test case from niteoweb.windmill:</p>
<pre>import unittest<br />from niteoweb.windmill import WindmillTestCase<br />from Products.PloneTestCase.setup import setupPloneSite<br />from Products.PloneTestCase.layer import onsetup<br />from Products.Five.zcml import load_config<br />from Testing import ZopeTestCase as ztc<br /><br />@onsetup<br />def load_zcml():<br />    import plone.app.skineditor<br />    load_config('configure.zcml', plone.app.skineditor)<br />    ztc.installPackage('plone.app.skineditor')<br /><br />load_zcml()<br />setupPloneSite(products=['plone.app.skineditor'])<br /><br />class SkinEditorIntegrationTestCase(WindmillTestCase):<br /><br />    def afterSetUp(self):<br />        """Setup for each test<br />        """<br />        ztc.utils.setupCoreSessions(self.app)<br />        self.setRoles(['Manager'])<br />        self.login_user()<br /><br />    def test_customize_logo(self):<br />        import pdb; pdb.set_trace()<br /><br />def test_suite():<br />    return unittest.defaultTestLoader.loadTestsFromName(__name__)</pre>
<p>(Most of this is standard test setup boilerplate. I could probably be a bit more sophisticated about the test setup and use layers or something, but this is the tried and true test setup I've been using since Martin published it in his book. The load_zcml method and setupPloneSite method are deferred and run when the Plone test layer is set up by the test runner.)<br /><br />So far I just added one test that enters pdb. At this point, I can run the test with bin/test -s plone.app.skineditor, and Windmill will start up the ZServer with a dummy Plone site, fire up a browser and a controller window, and then pause at the pdb. (Unlike Selenium, I don't have to install a browser plugin or have another process running first. Nice!) Then I can play around with the controller and start recording tests.  The afterSetUp method runs before each test, and calls niteoweb.windmill's "login_user" helper, which does an initial login to the site as a Manager user.  It also tells the test infrastructure to support sessions, which are needed for my app.<br /><br />Actually recording a test is mostly a point-and-click affair—hit the "record" button and go at it—but with a few caveats that I'll note below. It took a little bit for me to get used to the ways of selecting parts of the page and making assertions, but at least Windmill provides some flexibility. You can select by id, via XPath expressions, via JQuery selectors, or several other methods.  And assertions range from asserting that certain text is on the page to asserting that an arbitrary Javascript expression evaluates to true.<br /><br />Here's the full test case I ended up with (dumped from the controller using the "save" button):</p>
<pre>def test_customize_logo(self):<br />    client = self.wm<br />    client.click(id=u'user-name')<br />    # load customizer<br />    client.click(link=u'Site Setup')<br />    client.waits.forPageLoad(timeout=u'20000')<br />    client.click(link=u'Theme Editor')<br />    # go into advanced mode and customize based on the logo in the<br />    # non-active "plone_images" layer (Windmill doesn't do file uploads)<br />    client.click(link=u'Advanced')<br />    client.click(id=u'plone-app-skineditor-name-field')<br />    client.type(text=u'logo', id=u'plone-app-skineditor-name-field')<br />    client.click(id=u'plone-app-skineditor-filter-button')<br />    client.waits.forElement(timeout=u'', id=u'plone-app-skineditor-browser')<br />    client.click(xpath=u"//a[@id='skineditor-logo.png']/dt")<br />    client.waits.forElement(xpath=u"//dd[@class='plone-app-skineditor-layers']")<br />    client.click(jquery=u"('a[href*=plone_images/logo.png/manage_main]')[0]")<br />    client.waits.forElement(jquery=u"('#pb_1 input[value=Customize]')")<br />    client.click(name=u'submit')<br />    client.waits.forElement(timeout=u'', id=u'pb_2')<br />    # now reload and make sure the logo has the height we expect from the<br />    # customized image<br />    client.refresh()<br />    client.asserts.assertJS(js=u"$('#portal-logo').height() == 57")<br />    # now remove the customization<br />    client.click(id=u'plone-app-skineditor-name-field')<br />    client.type(text=u'logo', id=u'plone-app-skineditor-name-field')<br />    client.click(id=u'plone-app-skineditor-filter-button')<br />    client.waits.forElement(timeout=u'', id=u'skineditor-logo.png')<br />    client.click(xpath=u"//a[@id='skineditor-logo.png']/dt")<br />    client.waits.forElement(xpath=u"//dd[@class='plone-app-skineditor-layers']")<br />    client.click(link=u'Remove')<br />    # and confirm we're back to the original height<br />    client.refresh()<br />    client.asserts.assertJS(js=u"$('#portal-logo').height() == 56")</pre>
<p>And here's what it looks like to run that. (No clicking on my part involved!) The excitement starts about 0:17...</p>
<p>
<object data="http://wglick.org/jingswfplayer.swf" height="349" id="scPlayer" type="application/x-shockwave-flash" width="600">
<param name="data" value="http://wglick.org/jingswfplayer.swf" />
<param name="quality" value="high" />
<param name="bgcolor" value="#FFFFFF" />
<param name="flashVars" value="containerwidth=600&containerheight=349&content=http://wglick.org/windmill.swf" />
<param name="allowFullScreen" value="true" />
<param name="scale" value="showall" />
<param name="allowScriptAccess" value="always" />
<param name="base" value="http://wglick.org/" />
<param name="src" value="http://wglick.org/jingswfplayer.swf" />
<param name="flashvars" value="containerwidth=600&containerheight=349&content=http://wglick.org/windmill.swf" />
<param name="allowfullscreen" value="true" />
</object>
</p>
<p><br />Finally, here are a few caveats I ran into while setting up my first test, to customize the logo using plone.app.skineditor, and some last observations.<br /><br /></p>
<ul>
<li>Windmill can't do file uploads. This is a limitation of browser Javascript support / sandboxing, not of Windmill per se.  It would be nice if there were some command that would prime the Windmill HTTP proxy to add a particular file to the next HTTP request that comes through, so that uploads could at least be faked.</li>
<li>Windmill could be a bit smarter about handling AJAX requests when recording.  I needed to manually add a "waitForElement" step after each click that resulted in an AJAX load, to make sure that the load was complete before subsequent steps run.  It would be nice if there was a "wait for all ajax to complete" command so that I didn't have to identify a particular element to wait for.</li>
<li>I wish that there was a wrapper that would let me control Windmill using the same API as zope.testbrowser, to make it easier to convert existing tests to full browser tests.</li>
<li>So far I've only tried running tests in my default browser (Firefox), but it's possible to run in other browsers as well.  The niteoweb.windmill page on PyPI gives an example of a test layer for doing this.</li>
<li>It remains to be seen how cumbersome this sort of test will be to keep up-to-date as the product evolves.</li>
</ul>
<p>Overall Windmill provides a decent experience for recording tests and a really nice one for running them. Perhaps it is a tool we can use to do real browser testing for Plone core?<br /><br />If you've used Windmill and have any insights into how to use it effectively, I'd love to hear your thoughts in the comments.</p>]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2010-06-10T06:16:24Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/review-of-plone-3-products-development-cookbook">
    <title>Review of "Plone 3 Products Development Cookbook"</title>
    <link>http://davisagli.com/blog/review-of-plone-3-products-development-cookbook</link>
    <description>Packt Publishing recently released their new title, "Plone 3 Products Development Cookbook"</description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>Last month <a class="external-link" href="http://packtpub.com">Packt Publishing</a> released a new title to add their growing line of books on Plone development —<a class="external-link" href="https://www.packtpub.com/plone-3-3-products-development-cookbook/book"> <i>Plone 3 Products Development Cookbook</i></a> by Juan Pablo Giménez and Marcos F. Romero, reviewed by Martin Aspeli, Alec Mitchell, and Emanuel Sartor. The book aims to be a useful recipe-oriented resource for beginner- to intermediate-level developers of Plone add-on products. Having read the review copy Packt was kind enough to send me, I can recommend it as an up-to-date and more example-oriented complement to Packt's still-great established title on Plone 3 development, Martin Aspeli's <i><a class="external-link" href="https://www.packtpub.com/Professional-Plone-web-applications-CMS/book">Professional Plone Development</a></i>.<br /><br />I am a bit skeptical of the traditional cookbook format, in which a series of disjointed "recipes" are presented for dealing with various scenarios.  In practice these are most useful when your needs match the problem expressed in the recipe exactly, which is seldom. So I was glad to see that the authors of this cookbook have departed from that pattern somewhat. The book does contain a series of recipes, but they are all related to an overall project, to create a digital newspaper website, which runs throughout the book. This helps provide continuity and give clarity on which pieces of the included recipes are needed for the particular project as opposed to being general.<br /><br />The "recipe" concept is still evident in the book's format, though. Each chapter is divided into several tasks, each of which has 2 main parts: a "How to do it" section which lists the steps needed to complete the task, followed by a "How it works" section which describes the theory behind the task as well as the finer points of any of the steps which were non-obvious. It may seem a little weird that the explanation is separate from the instructions, but I think it could be a useful approach for a reader who learns well by example, and wants to try to figure out the meaning of the steps as they go before the answers are given. This format is one of the main things that sets this book apart from <i>Professional Plone Development</i>, which otherwise covers some similar topics.  I would recommend the latter to someone who learns best by reading first or who wants a reference, and this book to someone who learns best by doing first.<br /><br />In addition, this book is somewhat more up-to-date with current tools and techniques (though I anticipate that will be fully rectified in the <a class="external-link" href="http://www.martinaspeli.net/articles/professional-plone-development-plone-4-edition">upcoming Plone 4 edition of PPD</a> later this year).  Though the Products Development Cookbook goes into less depth in terms of explanation, the authors have been quite good about including links to up-to-date online resources such as the <a class="external-link" href="http://plonemanual.twinapex.fi/">Plone Developer Manual</a> where they don't have space to cover a subject in depth. The title, <i>Plone 3 Products Development Cookbook</i>, is a bit of a misnomer, by the way.  As far as I know, almost all the techniques described should work just as well in Plone 4.  (Unfortunately this repeats a mistake in naming from Erik Rose's <a class="external-link" href="https://www.packtpub.com/plone-3-for-education/book"><i>Plone 3 for Education</i></a> and Alex Clark's <a class="external-link" href="https://www.packtpub.com/plone-3-3-site-administration/book"><i>Plone 3 Site Administration</i></a>, which also both aim to be relevant for Plone 4...it is a shame Packt didn't realize it is possible to write a book that targets both Plone 3 and Plone 4.)<br /><br />I particularly enjoyed the chapters discussing content type creation. The book has examples of building an Archetype using ArgoUML and ArchGenXML, of building an Archetype using the 'archetype' ZopeSkel template, of creating a basic type using plone.app.content, and of creating a type using Dexterity...so this makes for an interesting quick comparison of the steps required in each case. (It is not an in-depth discussion of any of these approaches, though.  I would use the online manuals for Archetypes and Dexterity to learn the finer points of either system.)<br /><br />There are also several chapters that did a good job of recording some of the lore about Plone development tools and processes that has mostly been passed along in blogs and sprints so far.  For example, it covers the use of IPython and ipdb, plone.reload, in-browser testing with selenium, load testing with funkload, and a full production buildout including ZEO, varnish, pound, supervisor, and assorted utilities.  As a seasoned Plone developer myself, I even learned a few things from these sections. In addition to the topics mentioned above, the book also covers installing Plone, testing, internationalization, workflow and permissions, KSS, portlets, use of the Zope Component Architecture, and integrating with an external system (OpenX) via XML-RPC.<br /><br />For more information, orders, and errata, visit Packt Publishing's page for <a class="external-link" href="https://www.packtpub.com/plone-3-3-products-development-cookbook/book"><i>Plone 3 Products Development Cookbook</i></a>.  As usual for Packt, the book is available both in print and as an ebook.<br /><br /></p>]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2010-06-07T01:24:11Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/introducing-the-plone-resource-customizer">
    <title>Introducing the Plone resource customizer</title>
    <link>http://davisagli.com/blog/introducing-the-plone-resource-customizer</link>
    <description>Thanks to the sprint at Plone Symposium East, the Plone resource customizer is almost ready for a beta release.</description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>Going into the ZMI to edit templates and stylesheets and replace images sucks. Especially once we made it so there are two places you might need to go in Plone 3 (portal_skins and portal_view_customizations).  I've been working on a better way, the (drumroll please) Plone resource customizer<a class="anchor-link" href="#betternames">*</a>.</p>
<p>Thanks to the great work of my fellow sprinters at Plone Symposium East, I'm now ready to give a preview of the tool, which hopefully will see a first beta release real soon now™. Here's the screencast...<b><br /></b></p>
<p>
<object data="http://wglick.org/jingswfplayer.swf" height="349" id="scPlayer" type="application/x-shockwave-flash" width="600">
<param name="data" value="http://wglick.org/jingswfplayer.swf" />
<param name="quality" value="high" />
<param name="bgcolor" value="#FFFFFF" />
<param name="flashVars" value="thumb=http://content.screencast.com/users/davisagli/folders/Jing/media/ba6aedad-4c59-44cc-ba7d-d9a96bb3df48/FirstFrame.jpg&containerwidth=600&containerheight=349&content=http://wglick.org/customizer.swf" />
<param name="allowFullScreen" value="true" />
<param name="scale" value="showall" />
<param name="allowScriptAccess" value="always" />
<param name="base" value="http://wglick.org/" />
<param name="src" value="http://wglick.org/jingswfplayer.swf" />
<param name="flashvars" value="thumb=http://content.screencast.com/users/davisagli/folders/Jing/media/ba6aedad-4c59-44cc-ba7d-d9a96bb3df48/FirstFrame.jpg&containerwidth=600&containerheight=349&content=http://wglick.org/customizer.swf" />
<param name="allowfullscreen" value="true" />
</object>
</p>
<p>I forgot to mention in the screencast that Eric Steele has also started work on integrating this tool with Gloworm, so that it will be possible to find a resource to customize just by pointing and clicking (a la Firebug).</p>
<p>Currently the customizer displays items from CMF skin layers, browser view templates, viewlets, and portlets. Support for other things like browser resources or ZMI pages could probably be added, at least in a read-only fashion. The infrastructure is flexible enough to support new ways of registering resources that haven't been invented yet.</p>
<p>For now, if you're adventurous and want to try out the customizer, get a copy of the Plone 4 coredev buildout, and run it using -c experimental/skineditor.cfg</p>
<p>We are tracking bugs and ideas for improvement in the Plone bug tracker, with the component set to "Skin Editor."</p>
<p>This is only the beginning...David Bain is working on the customizer as part of the Google Summer of Code, and we've got lots of ideas about how to make it even more useful.</p>
<p><a name="betternames"></a>*better names hereby solicited :)</p>]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    
      <dc:subject>sprint</dc:subject>
    
    
      <dc:subject>customizer</dc:subject>
    
    <dc:date>2010-06-02T05:50:00Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/be-vigilant-know-your-buildout-threat-level">
    <title>Be vigilant: Know your Buildout Threat Level</title>
    <link>http://davisagli.com/blog/be-vigilant-know-your-buildout-threat-level</link>
    <description>The Buildout Threat Level indicator reports the percentage of buildouts that have succeeded in the past 4 hours.</description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>For those who missed my announcement last week on Twitter –</p>
<p>As a result of a mostly tongue-in-cheek conversation on IRC with Elizabeth Leddy, Alex Clark, and Matthew Wilkes, I decided to set up the global <a class="external-link" href="http://buildthreat.appspot.com">zc.buildout threat level indicator</a>.</p>
<p>The indicator collects reports of success and failure from actual buildout runs, and displays the current threat level based on the percentage of buildouts that have succeeded in the past 4 hours.</p>
<p>I'm happy to report that the current threat level is LOW.</p>
<p><img src="../images/Screenshot20100509at8.47.21PM.png" alt="Buildout Threat Level: Low" class="image-inline" /></p>
<p>Creating the indicator was mostly an excuse to try writing a Google App Engine app, but it's already proved its utility.  Last Thursday the threat level reached ELEVATED after a new distribute release started causing buildouts to fail under Python 2.4. Thanks to Tarek Ziadé for the quick response with a new distribute release to fix that.</p>
<p>You can help make the indicator more accurate!  Just add the <a class="external-link" href="http://pypi.python.org/pypi/buildout.threatlevel">buildout.threatlevel</a> extension to your buildout:</p>
<pre>[buildout]<br />extensions = buildout.threatlevel<br /></pre>
<p>This will display the current threat level as one of the initial steps when you run buildout (so that you have an opportunity to abort if it has reached SEVERE), and will ping my app engine app when the buildout is done to report success or failure. (No data is collected in this ping aside from a boolean indicating success, time, and IP address.)</p>
<p>Code for the GAE app and buildout extension is <a class="external-link" href="http://svn.plone.org/svn/collective/buildout/buildout.threatlevel">in the collective</a>.</p>]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2010-05-10T04:09:15Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/report-out-from-the-seattle-getpaid-recurring-payment-mini-sprint">
    <title>Report-out from the Seattle GetPaid recurring payment mini-sprint</title>
    <link>http://davisagli.com/blog/report-out-from-the-seattle-getpaid-recurring-payment-mini-sprint</link>
    <description>Six Plone developers in Seattle gathered this past weekend to work on adding support for recurring payments to PloneGetPaid.</description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p><a class="external-link" href="http://groundwire.org">Groundwire</a> organized a mini-sprint this past Thursday-Saturday to work on adding support for handling recurring subscription-based payments using <a class="external-link" href="http://plonegetpaid.com">PloneGetPaid</a>.  I had the honor of coordinating this work, and am happy to report that we had a very productive sprint and surpassed my goals for what we should accomplish.<br /><br />We added support for creating a recurring payment subscription to the Authorize.net and Paypal payment processors.  This will be done instead of a standard one-time payment if the cart contains a "recurring line item," with a corresponding interval and total # of payments.  (Carts containing multiple recurring items, or a mixture of recurring and non-recurring items, are not supported.)<br /><br />We also created two ways to get recurring line items into the cart.  The first way is similar to the way that other types of payment are configured in GetPaid: Add a marker to an arbitrary piece of content by selecting "Make recurring payment" from the action menu, and then enter the price, interval, and total # of payments.  (This can be done as long as the current payment processor supports recurring payments.)  Once an item is configured as a recurring payment, a portlet displays with an "Add to Cart" button that can be used to being purchasing.<br /><br />The second way to get recurring line items into the cart is via a new package, pfg.donationform, which provides a shortcut to set up a PloneFormGen form for making a one-time *or* recurring donation.  (Aside from the recurrence part this was possible before using getpaid.formgen, but the new package streamlines the setup.)  Selecting "Donation Form" from Plone's Add menu gives the site manager an opportunity to enter several settings which affect the creation of the standard PloneFormGen form.  The created form includes a custom "Donation Fieldset" which allows selecting from predefined donation levels or entering a custom amount, and specifying whether the payment should be one-time or recurring, and for how many months it should recur.  The donation form may also optionally be populated with fields to collect contact and billing info so that checkout can happen in a single step (at least with an on-site payment processor like the Authorize.net one).  When the donation form is submitted, line items will be added to the GetPaid cart based on the settings in the Donation Fieldset, and checkout will be initiated.</p>
<p><img src="../images/Screenshot20100509at8.12.31PM.png" alt="pfg.donationform" class="image-inline" /></p>
<p>Along the way, we also found time to add tests to getpaid.authorizedotnet and getpaid.formgen, which had no working tests before.<br /><br />All of this work has been on branches so far.  In the next week or so we'll be working on adding a few more tests, merging the changes, and making new releases of the relevant packages.  If you would like to review the work before that happens, please feel free.  Just check out the <a class="external-link" href="https://getpaid.googlecode.com/svn/getpaid.buildout/trunk">getpaid development buildout</a>, and build it using recurring-payment.cfg, to get the relevant branches.<br /><br />Thanks to the other sprinters and their employers for giving your time: Alex Tokar and Fulvio Casali of <a class="external-link" href="http://webcollective.coop">Web Collective</a>, Jesse Snyder of <a class="external-link" href="http://npowerseattle.org">NPower Seattle</a>, Cris Ewing of <a class="external-link" href="http://www.rad.washington.edu">U. of Washington Radiology</a>, and Rob Larubbio.  (And welcome aboard as GetPaid committers, Alex, Fulvio, Jesse, and Cris.)  Thanks also to <a class="external-link" href="http://ifpeople.net">ifPeople</a> and Juan Pablo Gímenez who did some work in 2008 that provided a helpful starting point for our work, to Zvezdan Petkovic from <a class="external-link" href="http://zope.com">Zope Corporation</a> for making new Zope 2.12-compatible releases of zc.ssl and zc.authorizedotnet, and to <a class="external-link" href="http://groundwire.org">Groundwire</a> for giving us a space to work and buying us lunch on Friday.</p>
<h3>Organizing a mini-sprint</h3>
<p>Some scattered thoughts on organizing an effective sprint:</p>
<ul>
<li>Sprints don't have to be large, expensive, or difficult to organize. If you have a few local Plonistas, a place to meet, and a clear shared goal, you can have a great sprint.  A small sprint means fewer brains, but increased communication and focus.</li>
<li>Not everyone participating in the sprint needs to have a deep familiarity with the product you're working on, but if you want to maximize productivity, everyone should have prior experience working on some Plone product.  And at least one person needs to have enough familiarity with the product and the sprint goal to produce a list of tasks ahead of time that can be easily divvied up amongst the participants.</li>
<li>A small amount of effort by the sprint coordinator in advance can make sure half the sprint isn't wasted on initial setup and bootstrapping tasks that don't benefit from more people.  Prior to the sprint, I made sure that Juan's previous branches were up-to-date against GetPaid trunk with passing tests (well, mostly).  I also made sure that all participants had commit access to the GetPaid repository, and tried to make sure that they ran the development buildout ahead of time.</li>
</ul>]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    
      <dc:subject>Groundwire</dc:subject>
    
    
      <dc:subject>getpaid</dc:subject>
    
    
      <dc:subject>sprint</dc:subject>
    
    <dc:date>2010-05-10T03:25:00Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/notes-on-migrating-this-blog-from-wordpress-to-plone">
    <title>Notes on migrating this blog from Wordpress to Plone</title>
    <link>http://davisagli.com/blog/notes-on-migrating-this-blog-from-wordpress-to-plone</link>
    <description>Includes example code for exporting blog posts and comments from Wordpress and importing them into Plone.</description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>I'm proud to be now running this blog on Plone 4, having just completed a migration from Wordpress.  Here are a few notes on the migration.</p>
<p>Initial steps that I won't go into detail on, at least for now:</p>
<ul>
<li>Obtained a <a class="external-link" href="http://linode.com">Linode</a> VPS with enough RAM to feed Plone (it's looking like my small Plone 4 site will need about 100MB to itself ... a greedy baseline, but that does include the database).</li>
<li>Created a Plone 4 buildout with ZEO and one Zope instance being run under supervisord.</li>
<li>Installed plone.app.caching, the new caching framework Martin Aspeli and Ric Newbery have been working on.  This still requires a number of svn checkouts at this point, but it's coming along nicely.</li>
<li>Installed <a class="external-link" href="http://plone.org/products/scrawl">Scrawl</a> so that I can manage blog entries as a separate content type.</li>
<li>Installed <a class="external-link" href="http://plone.org/products/plone.app.discussion">plone.app.discussion</a>, the new comment and discussion add-on created by Timo Stollenwerk, and its recaptcha add-on.  Made sure comments were enabled for the Blog Entry type, and that comment moderation was not turned on for the duration of the migration.</li>
<li>Turned off much of Plone's default HTML filtering.  Security is not a huge concern since I'm going to be the only one editing the site, and I tend to want to use fancy stuff in posts sometimes.</li>
</ul>
<h3 class=" ">Data Export</h3>
<p class=" ">Moving the posts and comments from my old blog's MySQL database was easier than I feared, though I did have to do a bit of coding.</p>
<p class=" ">I decided up front to do this by way of dumping the data to CSV files, rather than writing import code that read directly from MySQL. That was mostly a visceral reaction to a memory of a hard time getting MySQL-python installed and working properly once previously, and may have been an irrational fear.  But dumping the data from MySQL to CSV was easy enough, with the following two queries that grabbed just the data I needed:</p>
<pre class="mysql">SELECT ID, post_date, CONVERT(post_content USING latin1), post_title<br />INTO OUTFILE '/tmp/musings.csv'<br />FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\' LINES TERMINATED BY '\n'<br />FROM wp_posts, wp_post2cat<br />WHERE wp_posts.ID=post_id AND category_id=48<br />AND post_status='publish';<br /><br />SELECT comment_post_ID, comment_author, comment_author_email, comment_author_url, comment_date, CONVERT(comment_content USING latin1), user_id<br />INTO OUTFILE '/tmp/comments.csv'<br />FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\' LINES TERMINATED BY '\n'<br />FROM wp_comments<br />WHERE comment_approved = '1' AND comment_type='';</pre>
<p class=" ">This grabs a bunch of fields and dumps them into a CSV file with the given CSV dialect parameters.  The only tricky bit here is the call to CONVERT, which was needed because my raw data in MySQL had been improperly encoded.  A normal connection to MySQL defaults to the latin1 encoding (which is what MySQL calls windows-1252).  But Wordpress had been sending it utf8-encoded data, and the MySQL table had been configured to store things as utf8.  So when I stored data, MySQL was decoding the utf8 input as windows-1252, and then re-encoding as utf8.  On retrieval via a normal connection the reverse transformation was applied and it didn't matter, but SELECT INTO OUTFILE just copies the raw data from the table, which was effectively gobbledygook.  So I had to explicitly make MySQL convert the stored value to latin1 (read: decode it as utf8 and then encode as windows-1252) in order to end up with the utf8 I wanted.  This would have been needed for the other fields as well, except I wasn't using non-ASCII characters in them.</p>
<p class=" ">The category restriction on the first query makes sure that I only got the Plone-related posts from the old blog (I had been using it for personal blogging as well).  The comment type restriction on the second query excludes pingbacks.</p>
<h3 class=" ">Data Import</h3>
<p class=" ">I ended up writing this custom External Method to import the CSV data into Plone.  (I looked at transmogrifier, csvreplicata, and ArcheCSV, but for all of these it looked like I would have ended up writing a significant amount of code in the end anyway to get them to do what I wanted. And I knew I could do that "from scratch" in just a page or two of Python...)</p>
<pre class="python">import csv<br />import re<br />from DateTime import DateTime<br />from zope.component import queryUtility, createObject<br />from plone.i18n.normalizer.interfaces import IIDNormalizer<br />from plone.app.discussion.interfaces import IConversation<br /><br />PRE_RE = re.compile(r'(&lt;pre&gt;.*?&lt;/pre&gt;)', re.IGNORECASE | re.DOTALL)<br /><br />def cleanup_wordpress_text(text):<br />    text = PRE_RE.sub(lambda x: x.group(1).replace('\r\n\r\n', '\n\n'), text)<br />    return text.replace('\r\n\r\n', '&lt;p&gt;').replace('\r\n','\n').decode('utf-8')<br /><br />def importmusings(self):<br />    context = self<br />    reader = csv.reader(open('/tmp/musings.csv'), delimiter=',', quotechar='"', doublequote=False, escapechar='\\')<br />    posts = {}<br />    for row in reader:<br />        id, date, text, title = row<br />        short = queryUtility(IIDNormalizer).normalize(title)<br />        if short in context:<br />            del context[short]<br />        post = context[context.invokeFactory('Blog Entry', short)]<br />        post.setCreators(['davisagli'])<br />        post.setEffectiveDate(DateTime(date))<br />        post.setTitle(title)<br />        text = cleanup_wordpress_text(text)<br />        post.setText(text, mimetype='text/html')<br />        post.reindexObject()<br />        context.portal_workflow.doActionFor(post, 'publish')<br />        posts[id] = post<br /><br />    reader = csv.reader(open('/tmp/comments.csv'), delimiter=',', quotechar='"', doublequote=False, escapechar='\\')<br />    for row in reader:<br />        post_id, author, email, url, date, text, uid = row<br />        try:<br />            post = posts[post_id]<br />        except KeyError:<br />            continue<br />        conversation = IConversation(post)<br />        comment = createObject('plone.Comment')<br />        comment.text = cleanup_wordpress_text(text)<br />        if uid == '1':<br />            comment.creator = comment.author_username = 'davisagli'<br />            comment.author_name = 'David Glick'<br />            comment.author_email = 'dglick@gmail.com'<br />        else:<br />            comment.creator = None<br />            comment.author_name = author<br />            comment.author_email = email<br />        date = DateTime(date).asdatetime()<br />        comment.creation_date = comment.modification_date = date<br />        conversation.addComment(comment)<br /> <br />    return 'Done.'</pre>
<p class=" ">That cleanup_wordpress_text function turns double newlines from Wordpress into proper paragraph tags -- unless they're within a PRE tag.  The rest of the code is pretty readable -- yay, Python.</p>
<h3 class=" ">Syntax Highlighting</h3>
<p class=" ">You probably noticed one of the new site features -- syntax highlighting for blocks of code.  This is provided by the Pygments module, applied as a transformation to the entire response just before the Zope publisher returns it.  I achieved that via a plugin (called collective.pygmentstransform, available in the collective, and not released so far or probably ever) for Martin's plone.transformchain (also unreleased so far).  It's imperfect (not least because it guesses the language heuristically), but good enough for now I think.  Yes, I should probably be doing this as WSGI middleware, but I haven't spent the time to figure out how to run Zope 2.12 under WSGI yet.<br /><br /></p>]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    
      <dc:subject>plone4</dc:subject>
    
    
      <dc:subject>wordpress</dc:subject>
    
    
      <dc:subject>migration</dc:subject>
    
    <dc:date>2010-04-11T09:06:24Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/tahoe-sprint-off-and-running">
    <title>Tahoe sprint off and running</title>
    <link>http://davisagli.com/blog/tahoe-sprint-off-and-running</link>
    <description></description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[After carpooling together from San Francisco yesterday afternoon, the five participants in the <a href="http://www.coactivate.org/projects/tahoe-snow-sprint-2010/project-home">Tahoe Snow Sprint</a> arrived at our swanky lodgings overlooking Lake Tahoe.  The decor may be questionable, but there is more than enough space for us all, leaving us with crucial questions such whether to use the first floor or third floor bar at any given time.<p>We haven't done much coding yet, as much of the evening was spent with me giving a walkthrough of current Dexterity functionality, accompanied by much discussion of what works well and what doesn't, and what things we'd like to work on improving.<p>I will work on making various improvements to the usability and functionality of the through-the-web content type editor, hopefully with some help on UI design from Alex Limi.<p>Alex also hopes to work on designing some improved widgets.<p>Ross Patterson will work on adding support for Choice fields with vocabularies and sources to the TTW editor.<p>David Brenneman will look into ways of allowing Archetypes content to reference Dexterity content.<p>And I believe Joel Burton will work on exporting content types created through the web to a full installable package (as opposed to simply exporting the FTI like you can do already via portal_setup).<p>So let the sprinting begin!<p>We are hanging out in #sprint on freenode, so feel free to stop by and say hello to the tahoebot.]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2010-03-16T08:59:29Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/reflections-on-pycon-2010">
    <title>reflections on PyCon 2010</title>
    <link>http://davisagli.com/blog/reflections-on-pycon-2010</link>
    <description></description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[I just got back from the US PyCon 2010, my first Python Conference, where I had a blast. The conference felt to me a lot like Plone conferences in spirit, only with a greater diversity of software projects and of course more people (a record attendance of ~1100). It was held at the Hyatt in downtown Atlanta and was a great success logistically. One success in the organization of the conference was the push to get more women to attend, which resulted in 11% female attendees, an increase over previous years which I hope will continue as a trend.<p>Some highlights of the talks I attended were:
<ul>
<li>"Building Leafy Chat, DjangoDose, and Hurricane: Lessons Learned on the Real-Time Web with Python" by Alex Gaynor – Introduced me to Orbited, Twisted, Redis, and other tools for building scalable, interactive websites.
<li>"Managing the world's oldest Django project" by James Bennett – I found myself drawing parallels between the evolution of Django and Ellington that James presented and that of Zope and Plone.  The Django community is learning the same lessons about testing and reusability that we have.
<li>"What every developer should know about database scalability" by Jonathan Ellis – good general overview of different strategies for replication and caching (focused on concepts rather than any particular software)
<li>"Powerful Pythonic Patterns" by Alex Martelli – philosophizing on software patterns and anti-patterns in the Python context
<li>"Demystifying Non-Blocking and Asynchronous I/O" by Peter A Portante – very helpful beginner-level overview
<li>"Unladed Swallow: fewer coconuts, faster Python" by Collin Winter – an update on the state of Unladen Swallow, which was approved for being merged into CPython during the language summit just before PyCon
<li>"Pynie: Python 3 on Parrot" by Allison Randal – This one was for fun...I might keep an eye on Pynie just to see how a language actually gets implemented.
<li>"How Python is guiding infrastructure construction in Africa" by Roy Hyunjin Han – Covered the use of Python for recognizing buildings in satellite imagery to help with planning development, etc.
<li>"Why not run all your tests all the time? A Study of continuous integration systems" by C. Titus Brown – Bottom line: "Use Hudson."
<li>the infamous Testing in Python BoF, which was a 3-hour lightning talk session organized one evening by the folks from Disney, complete with pizza, beer, heckling, and goats (the goat meme was introduced by Terry Peppers as an alternative to lolcats in slides, and ended up being adopted as a testing mascot).
<li>"Tests and Testability" by Ned Batchelder – Not a lot new here for me, but a good overview by the creator of coverage.py.
</ul><p>Selecting which talk to go to was sometimes excruciating, and I'm looking forward to catching up with some of the ones I missed. Some of the ones I've heard recommended are:
<ul>
<li>"Deployment, development, packaging, and a little bit of the cloud" by Ian Bicking
<li>"The state of Packaging" by Tarek Ziadé
<li>"Scaling your Python application on EC2" by Jeremy Edberg – learnings from reddit
<li>"Dude, Where's My Database?" by Eric Florenzano
<li>"Understanding the Python GIL" by David Beazley – the hot topic of the conference
<li>"The Python and the Elephant: Large Scale Natural Language Processing with NLTK and Dumbo" by Nitin Madnani and Jimmly L. Lin
</ul><p>Videos of the talks are, amazingly, <a href="http://pycon.blip.tv">already becoming available</a>.  Kudos to the A/V team.<p>On Sunday my attention waned and I got a bit mischievous.  The Eldarion guys, who created <a href="http://typewar.com/">Type War</a>, set up <a href="http://pycon.ohwar.com/">OHWar</a>, a type war clone where you compete to correctly guess who said various quotes that were overheard at PyCon.  After playing for far too long and still failing to stay in first place for long, I decided it was a job for Python and created <a href="http://bitbucket.org/davisagli/ohwar-bot/src/tip/ohwar.py">an OHWar-playing bot</a>.  I left it running in screen and came back a few hours later to find that I had not only <a href="http://pycon.ohwar.com/stats">topped the leaderboard</a> but also hit the game's built-in score limit. :) This was also the evening that David Brenneman and I found the Django Pony unattended and added some "enhancements." ;)<p><img src="http://wglick.org/django_pony.jpg" alt="Django pony with Plone stickers" width="500" /><p>Zope and Plone were not very visible in the conference schedule (there was one talk on Plone GetPaid and Satchmo, one on using Plone with Salesforce in which I contributed a few minutes of technical material to go with Chris Johnson's high-level overview, and one on the interface/adapter concepts...as well as a couple relating to repoze.bfg which has a Zopish ancestry).  On the other hand, I believe Plone was, surprisingly, the only open source project with a booth in the exhibition hall.  We had a nice-looking display with the Plone banner that continues to be passed around to US events, a bunch of collateral and books for display, and a big monitor for demoing Plone 4. Various people took turns staffing the booth, including members of the Atlanta Plone group, and Chris Calloway for much of Saturday.  The Plone Foundation also subsidized World Plone Day T-shirts which a bunch of us wore on Saturday.  We gathered for a photo and ended up with around 30 people.<p><img src="http://pingfmmedia.s3.amazonaws.com/img/Q6utB5UC/rGHNthzuxAdICvlx.jpg" alt="Plone folks at PyCon" width="500" /><p>During the conference, a highlight for me was meeting and eating meals with various luminaries, including Jason Huggins (of Selenium fame), Holger Krekel (founder of PyPy), Wesley Chun (author of Core Python Programming) and even Guido himself (well, way down at the other end of the table).  I also got to interact briefly with Allison Randal (from the Perl community), while trying out and submitting a new test for pynie, a nascent Python implementation for the Parrot VM. I also now have a face to put with many additional names that I had only seen online before.<p>I was only able to join the sprints for one day, and mostly spent my time working on some miscellaneous tasks I hadn't been getting too.  However we were able to have a good meeting of GetPaid folks, to try to determine how to move forward with Brandon Rhodes' work to clean up payment processor configuration.  I also did some refactoring of the GetPaid development buildout to clean it up, make sure it still works, and pave the way for updating the product for compatibility with Plone 4.  If I had been able to stay longer, I think it would have been fun to participate in the great work being done in the Python packaging sprint, led by Tarek Ziadé and the <a href="http://twitter.com/packagingpig">Packaging Pig</a>.  Next year I will have to be sure to attend the entire sprint.
]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2010-02-23T22:50:47Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/reflections-on-building-a-member-directory-using">
    <title>Reflections on building a member directory using Plone and Salesforce.com</title>
    <link>http://davisagli.com/blog/reflections-on-building-a-member-directory-using</link>
    <description></description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[I promised Chris Johnson that I would write up some of my learnings from a project integrating <a href="http://plone.org">Plone</a> and <a href="http://salesforce.com">Salesforce.com</a>, which
<a href="http://groundwire.org">Groundwire</a> is just finishing up.  So here you go, Chris!<p>The goal of the project is to provide web access to a directory of businesses who have paid for membership and inclusion in our client's directory -- while keeping the master data for the directory within Salesforce.com, not Plone.  This involves several crucial challenges:
<ol>
 <li>How to present views for searching and browsing the Salesforce directory data within Plone
 <li>How to provide the ability for businesses to log in and update their member profile
 <li>How to provide the ability for businesses to apply and complete payment for membership, as well as to renew membership each year.
</ol><p>In this article, I'm going to focus on explaining how I approached the first two challenges.  This is much more of a hand wave in the right direction, assuming a fair amount of background in Plone, than a detailed tutorial. That said, feel free to ask me questions about aspects of the implementation that I gloss over here.<p><h2>Exposing the directory within Plone</h2><p>Querying Salesforce directly on each request is a non-starter for many use cases.  That's because Salesforce puts a pretty low limit on the number of API requests allowed per day (something like 1000 per user license).  This means that we need a way to mirror data from Salesforce within Plone, and then update it in batch (thereby using fewer API requests) every night.  (Building the directory as VisualForce pages within Salesforce Sites would be a valid alternative in some cases -- though requiring more work to integrate visually.  But for this project it was a requirement that we be able to store additional data such as logos within Plone, as well as link to related content items for a business.)<p>How do we model data from Salesforce within Plone?  It depends on what you need to do with the content in Plone.  If you just need to be able to search and display a listing of results, then there is no reason to create full-fledged content items.  In the past, for a case like this, I have just created temporary stub objects during a nightly dump of data from Salesforce, indexed them in a custom catalog, and then discarded the stubs.  This is the most lightweight option; you have a catalog full of data for building your search views, but no unnecessary data hanging around.<p>If you actually need to be able to navigate to a full page view of a particular directory item, then you probably need an actual content item.  I think Dexterity would be promising for this sort of thing, but for the project I'm just now wrapping up, I used Archetypes because I needed image scaling and the ability to link to other AT content as related items, both of which Dexterity doesn't have great support for yet.<p>Note that you don't actually need to define most aspects of the schema, if there are fields you want to display but don't need to have editable within Plone.  For example, my schema looks something like this:<p><pre>
MemberProfileSchema = document.ATDocumentSchema.copy() + atapi.Schema((
    atapi.TextField('sf_id'),
    atapi.TextField('mailingAddress'),
    # etc...
))
# hide most fields
for field in MemberProfileSchema.fields():
    if field.schemata == 'default' and field.__name__ not in ('text',):
        field.widget.visible = {'edit':'invisible', 'view':'visible'}
</pre><p>Fields like mailingAddress get populated during the nightly data dump, but don't appear on the edit form if you edit the member profile.  Why not?  Well, mostly because I figured it would be hard to get an Archetypes edit form to save things to Salesforce as well as Plone.  Alex Tokar at <a href="http://webcollective.coop">Web Collective</a> tells me he has successfully taken this approach, though.<p>Here is an abbreviated version of the browser view that is called once a night to pull in the data from Salesforce:<p><pre>
"""
SFDC sync view. This is intended to be run via cron every night to update
the member profiles based on data from Salesforce.com.

It will:

 * Find all Accounts with a member status of 'Current' or 'Grace Period' (in
   our client's Salesforce schema this is a custom rollup field based on various
   criteria).
 
 * For each Account, find an existing Member Profile object in Plone whose
   'sf_id' field value equals the Id of the Account, and update it.
   
 * Or, if no existing Member Profile was found, create a new one and publish it.

 * Retract any existing Member Profiles that were no longer found as Accounts
   with the Active or Grace Period membership status in Salesforce, so they are
   still present but not publicly visible.

"""

import logging
import transaction
from zope.component import getUtility
from Products.Five import BrowserView
from Products.CMFCore.utils import getToolByName
from plone.i18n.normalizer.interfaces import IIDNormalizer
from Products.CMFPlone.utils import safe_unicode
from Products.CMFPlone.utils import _createObjectByType

SOBJECT_TYPE = 'Account'
FIELDS_TO_FETCH = (
    'Id',
    'Name',
    'Description',
    'BillingStreet',
    'BillingCity',
    'BillingState',
    'BillingPostalCode',
    # etc...
    )
FETCH_CRITERIA = "Member_Status__c = 'Current' OR Member_Status__c = 'Grace Period'"
DIRECTORY_ID = 'directory'
PROFILE_PORTAL_TYPE = 'Member Profile'

logger = logging.getLogger('SFDC Import')

class UpdateMemberProfilesFromSalesforce(BrowserView):
    
    def __init__(self, context, request):
        BrowserView.__init__(self, context, request)
        self.catalog = getToolByName(self.context, 'portal_catalog')
        self.wftool = getToolByName(self.context, 'portal_workflow')
        self.normalizer = getUtility(IIDNormalizer)
    
    def getDirectoryFolder(self):
        portal = getToolByName(self.context, 'portal_url').getPortalObject()
    
        # create the directory folder if it doesn't exist yet
        try:
            directory = portal.unrestrictedTraverse(DIRECTORY_ID)
        except KeyError:
            _createObjectByType('Large Plone Folder', portal, id=DIRECTORY_ID)
            directory = getattr(portal, DIRECTORY_ID)
        
        return directory
    
    def findOrCreateProfileBySfId(self, name, sf_id):
        res = self.catalog.searchResults(getSf_id = sf_id)
        if res:
            # update existing profile
            profile = res[0].getObject()
            logger.info('Updating %s' % '/'.join(profile.getPhysicalPath()))
            return profile
        else:
            # didn't match sf_id or UID: create new profile
            name = safe_unicode(name)
            profile_id = self.normalizer.normalize(name)
            directory = self.getDirectoryFolder()
            profile_id = directory.invokeFactory(PROFILE_PORTAL_TYPE, profile_id)
            profile = getattr(directory, profile_id)
            profile.setSf_id(sf_id)
            profile.reindexObject(idxs=['getSf_id'])
            logger.info('Creating %s' % '/'.join(profile.getPhysicalPath()))
        
        return profile
    
    def updateProfile(self, profile, data):
        profile.setSf_id(data.Id)
        profile.setTitle(data.Name)
        if not profile.getText():
            profile.setText(data.Description, mimetype='text/x-web-intelligent')
        profile.setMailingAddress("%s\n%s, %s %s" % (data.BillingStreet, data.BillingCity,
                                                     data.BillingState, data.BillingPostalCode))
        # etc...
        
        # publish and reindex
        try:
            self.wftool.doActionFor(profile, 'publish')
        except:
            pass
        profile.reindexObject()
    
    def hideProfileBySfId(self, sf_id):
        res = self.catalog.searchResults(getSf_id = sf_id)
        profile = res[0].getObject()
        try:
            self.wftool.doActionFor(profile, 'reject')
        except:
            pass

    def queryMembers(self):
        """ Returns an iterator over the records of active members from Salesforce.com """
        sfbc = getToolByName(self.context, 'portal_salesforcebaseconnector')
        where = '(' + FETCH_CRITERIA + ')'
        soql = "SELECT %s FROM %s WHERE %s" % (
            ','.join(FIELDS_TO_FETCH),
            SOBJECT_TYPE,
            where)
        logger.debug(soql)
        res = sfbc.query(soql)
        logger.info('%s records found.' % res['size'])
        for member in res:
            yield member
        while not res['done']:
            res = sfbc.queryMore(res['queryLocator'])
            for member in res:
                yield member
    
    def __call__(self, queryMembers=queryMembers):
        """ Updates the member directory based on querying Salesforce.com """
        
        # 0. get list of sf_ids for the profiles we already know about, so we
        # can keep track of which ones we need to make private
        sf_ids_not_found = set(self.catalog.uniqueValuesFor('getSf_id'))
        
        # 1. fetch active Member Profile records, update ones that match,
        #    and create new ones
        for i, data in enumerate(queryMembers(self)):
            profile = self.findOrCreateProfileBySfId(name = data.Name, sf_id = data.Id)
            self.updateProfile(profile, data)
            
            # commit periodically (every 10) to avoid conflicts
            if not i % 10:
                transaction.commit()
            
            # keep track of which profiles we need to hide
            try:
                sf_ids_not_found.remove(data.Id)
            except KeyError:
                pass
        
        # 2. hide any profiles that are no longer active
        for sf_id in sf_ids_not_found:
            self.hideProfileBySfId(sf_id)
</pre><p>All that's left is writing the view which actually queries the catalog for these member profiles and presents them as a listing, which is relatively straightforward, and left as an exercise for the reader. :)<p><h2>Allowing updates to directory profiles</h2><p>So if the Archetypes content type doesn't allow edits to most of its fields,
how did I provide for logged-in members to edit profile info?  Well, there are 2 parts:<p><ol>
 <li>The <a href="http://plone.org/products/salesforceauthplugin">Salesforce Auth Plugin</a> allows
     logins to Plone based on Account records in Salesforce (by matching on custom username and password fields on the Account).
 <li>A custom z3c.form form reads values from the Account associated with the currently logged-in user, and writes to both that Account record in Salesforce and also to the associated Member Profile archetype within Plone (so that updates appear in the directory immediately).
</ol><p>I won't go into detail on the configuration of the Auth Plugin, as it is covered in the package's documentation.  I configured it to load the Salesforce Id of the Account and several other fields into PAS member properties, for easy access within Plone.  I did not configure all of the account fields as member properties -- while I could have done so, I didn't see much utility in that, since Plone can't (at least not yet) automatically generate an edit form for all the member properties.<p>Instead, I built a custom z3c.form form that reads and writes directly to Salesforce.  This turned out to be less complicated than I anticipated, mostly thanks to a new ORM-style library I built for wrapping the objects returned from Salesforce by beatbox (with attributes corresponding to Salesforce field names) with a model whose attribute names match the field names of the form schema -- allowing use of the wrapper as the context of a z3c.form form. I'm not yet going to post the implementation of this library, as I intend to make some significant changes to the API before releasing it (real soon now?). But let me at least show you what using it looks like (again I have simplified from the real code):<p><pre>
from zope.interface import implements
from z3c.form import form, field, button
from plone.z3cform.layout import wrap_form
from plone.memoize.instance import memoize
from Products.CMFCore.utils import getToolByName

from sforzando import SFObject, SFField

class IAccountGeneralInfo(Interface):
    """ Schema for member profile edit form """
    business_name = schema.TextLine(title = u'Business Name')
    # etc...

class SFAccount(SFObject):
    """ Adapts a Salesforce Account to the profile edit form schema"""
    implements(IAccountGeneralInfo)
    
    _sObjectType = 'Account'
    
    sf_id = SFField('Id')
    business_name = SFField('Name')
    # etc...

class ProfileEditForm(form.Form):
    """ An edit form for the current authenticated member's Account """
    
    label = u'Update Profile'
    fields = field.Fields(IAccountGeneralInfo)

    def _get_sf_id(self):
        """ Find the Salesforce Account Id corresponding to the current logged in member. """
        mtool = getToolByName(self.context, 'portal_membership')
        member = mtool.getAuthenticatedMember()
        sf_id = member.getProperty('sf_id')
        if not sf_id:
            raise Exception("Did not find valid Salesforce ID for member '%s'" % member.getId())
        return sf_id

    @memoize
    def getContent(self):
        """ Provides the object this form will edit.
            Memoized so we always get the same one for a given request. """
        sfbc = getToolByName(context, 'portal_salesforcebaseconnector')
        return SFAccount(sfbc, "Id='%s'" % self._get_sf_id())

    @button.buttonAndHandler(u'Update Profile')
    def handleUpdate(self, action):
        """ Handler for the Update Profile button """
        data, errors = self.extractData()
        if not errors:
            self.status = u'Changes saved.'
            # save changes to Salesforce
            sf_id = self._get_sf_id()
            sfbc = getToolByName(context, 'portal_salesforcebaseconnector')
            SFAccount.update(sfbc, id=sf_id, **data)
            # etc...additional code to update the local AT-based copy of the Account data...

ProfileEditView = wrap_form(ProfileEditForm)
</pre><p>Formlib would probably also work just as well as z3c.form.  And certainly using a PloneFormGen form with the 'update' feature of the <a href="http://plone.org/products/salesforcepfgadapter">salesforcepfgadapter</a> would work without need for coding, if you don't need a particularly fancy form.  As long as you mapped the Salesforce object Id as a member property in the Auth Plugin configuration, it's pretty easy to use that as the basis for determining which object the form should edit.<p><h2>In conclusion</h2><p>I'm pretty excited about the results of this project, which is one of the deeper integrations of Plone and Salesforce.com that I have worked on, and which builds on the tools <a href="http://groundwire.org">Groundwire</a> has led the development of over the past few years -- especially the Salesforce Auth Plugin.  Giving Plone the ability to accept logins based on a CRM system opens the door to a lot of exciting possibilities -- think about being able to show visitors targeted content based on what your database knows about their interests or location, or allowing them to share content with other visitors from the same geographic area.<p>If you are putting to good use the tools and code discussed here, or are finding other cool things to do by integrating Plone and Salesforce, I'd love to hear about it.]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2010-02-09T00:20:05Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/using-haproxy-with-zope-via-buildout">
    <title>Using HAProxy with Zope via Buildout</title>
    <link>http://davisagli.com/blog/using-haproxy-with-zope-via-buildout</link>
    <description></description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[After my <a href="http://david.wglick.org/2010/on-zope-multiple-cores-and-the-gil">post on reducing GIL contention by using fewer Zope threads</a>, Lee Joramo asked for more information on setting up <a href="http://haproxy.1wt.eu">HAProxy</a>, so let me share my configuration.  Much of the credit for this goes to Hanno Schlichting and Alex Clark, who offered me much good advice and a sample configuration, respectively.<p>First, a few words about what HAProxy offers.  For the past couple years I've been using Pound to load balance between multiple backend Zope instances.  But recently I've been hearing recommendations from people I trust (such as Jarn and Elizabeth Leddy) to try HAProxy instead.<p>HAProxy offers some nice features:
- Backend health checks
- Various load-balance algorithms for how requests get distributed to backends
- Can do sticky sessions so that an authenticated user always hits the same backend
- Warmup time (don't send as many requests to a Zope instance while it's starting up)
- Provides a status page giving info on backend status and uptime, # of queued requests, # of active sessions, # of errors, etc.<p>Some of these are possible with pound too, but the status screen was really the "killer app" for me.  This is fun to watch but also very useful for doing rolling restarts when new code needs to be deployed without an interruption in service.<p><img src="http://wglick.org/haproxy_status.png" alt="HAProxy status page" /><p><h2>Configuration</h2><p>In my buildout.cfg I added:<p><pre>
[buildout]
...
parts =
    ...
    haproxy-build
    haproxy-conf

[haproxy-build]
recipe = plone.recipe.haproxy
url = http://dist.plone.org/thirdparty/haproxy-1.3.22.zip

[haproxy-conf]
recipe = collective.recipe.template
input = ${buildout:directory}/haproxy.conf.in
output = ${buildout:directory}/etc/haproxy.conf
maxconn = 24000
ulimit-n = 65536
user = zope
group = staff
bind = 127.0.0.1:8080
</pre><p>Here, we add a part called "haproxy-build" which uses the plone.recipe.haproxy recipe to build haproxy from source and add a bin/haproxy script for running it, and a part called "haproxy-conf" which builds the HAProxy configuration file by filling in variables in a template file called haproxy.conf.in.<p>Be sure to set the user and group variables to the user and group you want HAProxy to run as, and update the bind variable to set the port to which HAProxy should bind.<p>I run most of my Plone stack using <a href="http://supervisord.org/">supervisord</a>, so I also updated my supervisord configuration in buildout to run HAProxy:<p><pre>
[supervisor]
recipe = collective.recipe.supervisor
...
programs =
    ...
    10 haproxy ${buildout:directory}/bin/haproxy [ -f ${buildout:directory}/etc/haproxy.conf -db ]
</pre><p>In a real life deployment, you'll probably also want a caching reverse proxy like squid or varnish sitting in front of HAProxy.<p>What about the contents of haproxy.conf.in?  Here's mine:<p><pre>
global
  log 127.0.0.1 local6
  maxconn  ${haproxy-conf:maxconn}
  user     ${haproxy-conf:user}
  group    ${haproxy-conf:group}
  daemon
  nbproc 1

defaults
  mode http
  option httpclose
  # Remove requests from the queue if people press stop button
  option abortonclose
  # Try to connect this many times on failure
  retries 3
  # If a client is bound to a particular backend but it goes down,
  # send them to a different one
  option redispatch
  monitor-uri /haproxy-ping

  timeout connect 7s
  timeout queue   300s
  timeout client  300s
  timeout server  300s

  # Enable status page at this URL, on the port HAProxy is bound to
  stats enable
  stats uri /haproxy-status
  stats refresh 5s
  stats realm Haproxy\ statistics

frontend zopecluster
  bind ${haproxy-conf:bind}
  default_backend zope

# Load balancing over the zope instances
backend zope
  # Use Zope's __ac cookie as a basis for session stickiness if present.
  appsession __ac len 32 timeout 1d
  # Otherwise add a cookie called "serverid" for maintaining session stickiness.
  # This cookie lasts until the client's browser closes, and is invisible to Zope.
  cookie serverid insert nocache indirect
  # If no session found, use the roundrobin load-balancing algorithm to pick a backend.
  balance roundrobin
  # Use / (the default) for periodic backend health checks
  option httpchk

  # Server options:
  # "cookie" sets the value of the serverid cookie to be used for the server
  # "maxconn" is how many connections can be sent to the server at once
  # "check" enables health checks
  # "rise 1" means consider Zope up after 1 successful health check
  server  plone0101 127.0.0.1:${zeoclient1:http-address} cookie p0101 check maxconn 2 rise 1
  server  plone0102 127.0.0.1:${zeoclient2:http-address} cookie p0102 check maxconn 2 rise 1
</pre><p>This assumes that I have Zope instances built by parts called "zeoclient1" and "zeoclient2" in my buildout; you'll probably need to update those names.<p>You may want to adjust the "option httpchk" line to use a different URL for checking whether the Zope instances are up -- you want to point at something that can be rendered as quickly as possible (in my case it's the Zope root information screen, so I'm not too worried).<p>The maxconn setting for each backend should be at least the number of threads that that Zope instance is running. Laurence Rowe pointed out to me that it should probably not be set to 1, since Zope also serves some things (blobs and ) via file stream iterators, which happens apart from the main ZPublisher threads.  (So setting maxconn to 1 would mean serving a large blob could block other requests to that backend, for instance.)<p>See <a href="http://haproxy.1wt.eu/download/1.3/doc/configuration.txt">the HAProxy configuration documentation</a> for more details on the settings that can be used in this file.]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2010-01-19T23:14:53Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/on-zope-multiple-cores-and-the-gil">
    <title>on Zope, multiple cores, and the GIL</title>
    <link>http://davisagli.com/blog/on-zope-multiple-cores-and-the-gil</link>
    <description></description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[I recently installed HAProxy as a load-balancer for a site that had previously been running using a single Zope instance using 4 threads.  I switched to 2 instances using 2 threads each, load-balanced by HAProxy.  I wasn't anticipating that this change would have a noticeable effect on the site's performance, so was happily surprised when the client mentioned that users of the site were commenting on the improved speed.<p>But why did the site get faster?<p>Looking at a munin graph of server activity, I observed a noticeable drop in the number of rescheduling interrupts -- a change that coincided with my change in server configuration:<p><img src="http://wglick.org/interrupts.png" alt="graph showing decreased contention when I switched to more Zope instances with fewer threads" /><p>I suspect that the "before" portion of this graph illustrates a problem that occurs when running multi-threaded Python programs on multi-core machines, wherein threads running in different cores fight for control of the Global Interpreter Lock (a problem Dave Beazley has called to the community's attention <a href="http://www.dabeaz.com/python/GIL.pdf">in a recent presentation</a>) -- and that this explains the improvement in performance once I switched to multiple processes with fewer threads.  By switching to multiple processes, we let concurrent processing get managed by the operating system, which is much better at it.<p>Moral of the story: If you're running Zope on a multi-core machine, having more than 2 threads per Zope instance is probably a bad move performance-wise, compared to the option of running more (load-balanced) instances with fewer threads.<p>(Using a single thread per instance might be even better, although of course you need to make sure you have enough instances to still handle your load, and you need to make sure single-threaded instances don't make calls to external services which then call back to that instance and block.  I haven't experimented with using single-threaded instances yet myself.)
]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2010-01-18T00:12:43Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/come-improve-dexterity-at-the-tahoe-snow-sprint">
    <title>Come improve Dexterity at the Tahoe Snow Sprint</title>
    <link>http://davisagli.com/blog/come-improve-dexterity-at-the-tahoe-snow-sprint</link>
    <description></description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[This year the West Coast is hosting our own version of the infamous Snow Sprint. I'm really looking forward to spending a week coding, hanging out with Plonistas, and playing in the snow at the upcoming <a href="http://www.coactivate.org/projects/tahoe-snow-sprint-2010/project-home">Tahoe Snow Sprint</a>, organized by David Brenneman (dbfrombrc) and coming to California's Sierra Nevada this March 15-19.<p>The goal of the sprint is to improve the <a href="http://code.google.com/p/dexterity/">Dexterity</a> content type framework (a modern alternative to Archetypes created by Martin Aspeli and others).  As part of the Dexterity team, I want to offer the following list of potential projects to help get your creative juices flowing.<p>At the sprint, you could...
<hr />
Implement one of Dexterity's missing features, such as:
 <ul><li>a way to relate Dexterity content to Archetypes content and vice versa
 <li><a href="http://code.google.com/p/dexterity/issues/detail?id=72">automatic image scales</a>
 <li>"exclude from navigation" behavior
 <li><a href="http://code.google.com/p/dexterity/issues/detail?id=74">multilingual content</a>
 <li><a href="http://code.google.com/p/dexterity/issues/detail?id=8">link integrity checking</a>
 <li><a href="http://code.google.com/p/dexterity/issues/detail?id=49">selectable type constraints for containers</a>
 </ul><p>Fix some of the other <a href="http://code.google.com/p/dexterity/issues/list">outstanding issues</a> in the Dexterity issue tracker</a>.<p>Create a ZopeSkel template for Dexterity-based projects.<p>Improve the through-the-web content type editor.
 <ul><li>improve usability and/or sexiness
 <li>add UI for exporting types for work on the filesystem
 <li>add support for defining vocabularies
 <li>add support for selecting/configuring custom widgets
</ul><p>Create an editor that allows through-the-web editing of new behaviors (which can then be applied to existing types in a schema-extender-like fashion.)<p>Add a view editor to accompany the through-the-web schema editor.  Deco is coming and will be great, but in the meantime it would be nice to at least have something that generates a basic view template based on your schema and then lets you tweak it and export it to the filesystem.<p>Build a better workflow editor to accompany the above.<p>Write a guide to migrating Archetypes-based content types to Dexterity.  Or build a tool to do it automatically.<p>Create replacements for the ATContentTypes types using Dexterity types.<p>Determine how to handle existing content items sanely when editing schemas.<p>Devise a PloneFormGen successor that stores its schema in a fashion similar to Dexterity, and makes it easy to convert a form + results into a full-blown content type.  Bonus points if the form editing is done using Deco. :)
<hr />
There are so many interesting possibilities I'm having trouble deciding what to focus on myself.  Space is limited, so if any of this strikes your fancy, head on over to Coactivate and <a href="http://www.coactivate.org/projects/tahoe-snow-sprint-2010/sign-up-here">sign right up</a> to join us at the sprint!
]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2010-01-12T22:46:13Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>


  <item rdf:about="http://davisagli.com/blog/mr-igor">
    <title>mr.igor</title>
    <link>http://davisagli.com/blog/mr-igor</link>
    <description></description>
    <content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[Today I released <a href="http://pypi.python.org/pypi/mr.igor">mr.igor</a>, a utility for helping you write Python faster by filling in missing imports based on where you've imported the names from before.<p>Here's a one-minute screencast showing how it works.<p><object width="500" height="386"> <param name="movie" value="http://content.screencast.com/users/davisagli/folders/Jing/media/3a4ea936-9346-4597-9a4d-ba9d545db34b/jingswfplayer.swf" /></param> <param name="quality" value="high" /></param> <param name="bgcolor" value="#FFFFFF" /></param> <param name="flashVars" value="thumb=http://content.screencast.com/users/davisagli/folders/Jing/media/3a4ea936-9346-4597-9a4d-ba9d545db34b/FirstFrame.jpg&containerwidth=500&containerheight=386&content=http://content.screencast.com/users/davisagli/folders/Jing/media/3a4ea936-9346-4597-9a4d-ba9d545db34b/00000004.swf" /></param> <param name="allowFullScreen" value="true" /></param> <param name="scale" value="showall" /></param> <param name="allowScriptAccess" value="always" /></param> <param name="base" value="http://content.screencast.com/users/davisagli/folders/Jing/media/3a4ea936-9346-4597-9a4d-ba9d545db34b/" /></param>  <embed src="http://content.screencast.com/users/davisagli/folders/Jing/media/3a4ea936-9346-4597-9a4d-ba9d545db34b/jingswfplayer.swf" quality="high" bgcolor="#FFFFFF" width="500" height="386" type="application/x-shockwave-flash" allowscriptaccess="always" flashvars="thumb=http://content.screencast.com/users/davisagli/folders/Jing/media/3a4ea936-9346-4597-9a4d-ba9d545db34b/FirstFrame.jpg&containerwidth=500&containerheight=386&content=http://content.screencast.com/users/davisagli/folders/Jing/media/3a4ea936-9346-4597-9a4d-ba9d545db34b/00000004.swf" allowfullscreen="true" base="http://content.screencast.com/users/davisagli/folders/Jing/media/3a4ea936-9346-4597-9a4d-ba9d545db34b/" scale="showall"></embed> </object>]]></content:encoded>
    <dc:publisher>No publisher</dc:publisher>
    <dc:creator>David Glick</dc:creator>
    <dc:rights></dc:rights>
    <dc:date>2009-12-14T22:28:40Z</dc:date>
    <dc:type>Blog Entry</dc:type>
  </item>





</rdf:RDF>
