Strk's Blog

Hello, Burt!

Sat, 08 Feb 2020 14:15:18 +0000

Following our first album togheter, and after moving to Siena for taking formal music studies, me and Elisabetta have started a new project with a local guitarist and a new setup: she plays bass in this combo.

The bass was sold to her by Fabio, a friend of mine who wasn’t finding the time to play it. That friend was the inspiration for me to play the songs written by Burt Bacharach, who I did not know much about, before Fabio mentioned him to me.

Bacharach is known for his great melodies, which is why playing his music with a melodic instrument, like the trumpet, is so fun and satisfactory you would never stop. Nonetheless trumpet can be hard on your lips, so I’ve been also singing every now and then to let them rest a little bit 🙂

So, with the band ready, I’ve been producing lead sheets with lilypond and recording some of the pieces with ardour. Ain’t Free Software lovely ?

The result of such work can be heard live or on bandcamp, which also provides the embedded player in this post.

Enjoy!

Slivers of life

Fri, 04 Jan 2019 09:18:29 +0000

Frammenti by Blue Leafs

The first album of our jazz trio is finally completed. It took a few months to find the time to sit together in the wonderful tiny studio of Danilo, the pianist and producer. It didn’t help that me and Elisabetta, the singer, moved 3 hours away from the studio to attend a jazz accademy, following our hearts.

I’m playing the trumpet, a small and lovely instrument I’ve had the pleasure to start playing around spring 2017, blowing into it daily ever since. On the lake beach, in the archeological park, in the urban park, even in the botanical garden on the other side of the world.

My trumpet represents my getting back to myself, to my body, in connection to my soul. All computers and no music was making me a dull boy.

Trumpet is a real thing, a physical instrument, an amplifier of your lips vibration. Not a chip driven speaker with mathematically defined rules, but a physical laws obeying device. It taught me about harmonic series, about resonance, about my diaphragm and breathing.

Trumpet taught me to remember about myself, and soundly cure me.

To my experience, playing the trumpet is very similar to chanting mantras, it can have the same effects on the brain and the body.

Hoping you can enjoy some of it, happy listening !

Trac from Mutt

Wed, 11 Nov 2015 07:30:32 +0000

Dedicated to those who do not want to surrender to proprietary services for code and tickets management: integrating Cartman with Mutt.

The scenario is Trac instances that send email notifications on ticket activity, as is the case with the OSGeo ones (there’s been some recent discussion about the role of its infrastructure).

The Cartman command to comment on a ticket (say #33) of a project (say postgis) would be:

cm -s postgis 33

As you usually comment on some input received by mail (either a new ticket or another comment), building that command line requires eye-parsing for project name and ticket number, and then typing it on the shell.

Mutt allows you to define a macro to do all of that with a single keystroke. Here’s the macro I’ve written for me, to map -t to trac-reply.

macro pager \Ct \
 "<pipe-message>~/bin/muttcm --save\n<shell-escape>~/bin/muttcm\n"

The ~/bin/muttcm script saves the headers of the piped message when invoked with --save; finds the most recently saved headers, parse them for Trac coordinates and invokes Cartman with appropriate arguments.

The script in its current incarnation follows, but it is very likely that it will grow in the future to adapt more to my personal needs and preferences.

That’s what makes Free Software better than any shining proprietary service after all, isn’t it ?

UPDATE: I’ve given the code a dedicated page

#!/bin/sh
#
# Script integrating cartman (http://tamentis.com/projects/cartman/)
# with mutt (http://www.mutt.org/)
#
# Written (2015) by Sandro Santilli <strk@keybit.net>
# Released under Creative Commons Public Domain Dedication 1.0
# (https://creativecommons.org/publicdomain/zero/1.0/)
#
# Example macro to have <CTRL>-t trigger commenting on a ticket
# we're reading a mail notification of:
#
# macro pager \Ct "<pipe-message>~/bin/muttcm --save\n<shell-escape>~/bin/muttcm\n"
#
# The script expects to find a `cartman` configuration section
# matching the lowercased value of the X-Trac-Project header
# in the trac notification area.
#

TMPDIR=/tmp/

if test "$1" = "--save"; then
 TS=`date +'%Y%m%d%H%M%S'`
 FILE=${TMPDIR}/muttcm.in.${TS}
 umask 0077
 sed '/^$/q' > ${FILE}
 echo "Headers saved in ${FILE}"
 exit
fi

# find most recent file in ${TMPDIR}/muttcm.in.*
LASTFILE=`'ls' ${TMPDIR}/muttcm.in.* | tail -1`

PROJ=`grep '^X-Trac-Project:' "${LASTFILE}" | awk '{print $2}' | tr '[A-Z]' '[a-z]'`
TICK=`grep '^X-Trac-Ticket-ID:' "${LASTFILE}" | awk '{print $2}'`
rm ${LASTFILE}

if test -z "${PROJ}"; then
 echo "Cannot find an X-Trac-Project header"
 exit 1
fi
if test -z "${TICK}"; then
 echo "Cannot find an X-Trac-Ticket-ID header"
 exit 1
fi

echo 'PROJ: ' $PROJ
echo 'TICK: ' $TICK

cm -s "${PROJ}" comment "${TICK}"

On the fly simplification of topologically defined geometries

Fri, 08 Mar 2013 09:05:44 +0000

Keeping GIS data in full-resolution and simplifying it on demand is a known challenge: simplification has to be fast and its output has to be topologically consistent.

We saw how to get a topologically consistent simplified version of a full layer, but that method isn’t fast enough for on-demand usage. Also, we saw how to perform a fast simplification by sacrificing the degree or generalization so that the introduced inconsistency would not be visible on a rendering surface.

An approach balancing speed and quality would take advantage of the topological definition of geometries to constraint the simplification work by ensuring shared edges get an identical treatment.

This is now possible with the addition in PostGIS of a new version of ST_Simplify accepting a TopoGeometry as input.

In the picture above you can see the difference between using Geometry or TopoGeometry objects for simplifying the provinces of Tuscany with a tolerance of 8km. The ST_Simplify function runs at comparable speed in both cases.

GEOS 3.3.4 released

Thu, 31 May 2012 19:25:57 +0000

New bug fix and improvement release for the 3.3 series of GEOS is out.

Changes since 3.3.3:

Do not abort on NaN overlay input (#530)
Reduce CommonBitsRemover harmful effects during overlay op (#527)
Better cross-compiler support (#534)
Enable overlay ops short-circuits (#542)
Envelope-based short-circuit for symDifference (#543)
Fix support for PHP 5.4 (#513)
Fix TopologyPreservingSimplifier invalid output on closed line (#508)
Reduce calls to ptNotInList, greatly speeding up Polygonizer (#545)

Simplifying a map layer using PostGIS topology

Fri, 13 Apr 2012 12:30:42 +0000

Following a recent research about how to simplify a multipolygon layer while keeping topological relationships intact, here’s my take on that, using the PostGIS topological support.

The data

French administrative subdivisions, called “départements”, will be used. Data can be downloaded here.

It is composed by 96 multipolygons for a total of 47036 vertices.

Principle of simplification

We convert a layer’s Geometries to TopoGeometries
We simplify all edges of the built topology
We convert the (now-simplified) TopoGeometries back to Geometries

Steps

The following steps assume you loaded the shapefile into a table named “france_dept”.

-- Create a topology
SELECT CreateTopology('france_dept_topo', find_srid('public', 'france_dept', 'geom'));

-- Add a layer
SELECT AddTopoGeometryColumn('france_dept_topo', 'public', 'france_dept', 'topogeom', 'MULTIPOLYGON');

-- Populate the layer and the topology
UPDATE france_dept SET topogeom = toTopoGeom(geom, 'france_dept_topo', 1); -- 8.75 seconds

-- Simplify all edges up to 10000 units
SELECT SimplifyEdgeGeom('france_dept_topo', edge_id, 10000) FROM france_dept_topo.edge; -- 3.86 seconds

-- Convert the TopoGeometries to Geometries for visualization
ALTER TABLE france_dept ADD geomsimp GEOMETRY;
UPDATE france_dept SET geomsimp = topogeom::geometry; -- 0.11 seconds

The SimplifyEdgeGeom function

You may have noticed that the “SimplifyEdgeGeom” is not a core function. It is a function I wrote for the purpose of catching topological problems introduced by simplification.

The naive call would be:

SELECT ST_ChangeEdgeGeom('france_dept_topo', edge_id, ST_Simplify(geom, 10000))
FROM france_dept_topo.edge;

The problem with the above call is that any simplification introducing a topology error would be rejected by ST_ChangeEdgeGeom by throwing an exception and the exception would rollback the whole transaction leaving you with no edge changed. Possible topology errors introduced are: edges collapsing to points, intersecting self or other edges.

The SimplifyEdgeGeom function wraps the ST_ChangeEdgeGeom call into a subtransaction and handles exceptions by reducing the simplification factor until it succeeds. The version I used reduces simplification factor in half at each failure, dropping down to zero around 1e-8. You can roll your own with other heuristics or generalize this one to take parameters about stepping and limits.

Here’s the function:

CREATE OR REPLACE FUNCTION SimplifyEdgeGeom(atopo varchar, anedge int, maxtolerance float8)
RETURNS float8 AS $
DECLARE
  tol float8;
  sql varchar;
BEGIN
  tol := maxtolerance;
  LOOP
    sql := 'SELECT topology.ST_ChangeEdgeGeom(' || quote_literal(atopo) || ', ' || anedge
      || ', ST_Simplify(geom, ' || tol || ')) FROM '
      || quote_ident(atopo) || '.edge WHERE edge_id = ' || anedge;
    BEGIN
      RAISE DEBUG 'Running %', sql;
      EXECUTE sql;
      RETURN tol;
    EXCEPTION
     WHEN OTHERS THEN
      RAISE WARNING 'Simplification of edge % with tolerance % failed: %', anedge, tol, SQLERRM;
      tol := round( (tol/2.0) * 1e8 ) / 1e8; -- round to get to zero quicker
      IF tol = 0 THEN RAISE EXCEPTION '%', SQLERRM; END IF;
    END;
  END LOOP;
END
$ LANGUAGE 'plpgsql' STABLE STRICT;

Performance

The times shown near the “expensive” steps give you an indication of the performance you may expect. It’s about 13 seconds in total for the 3 steps outlined in the first paragraph.

A single run of the simplification step brought vertices down to 1369 (from 47036).

Timings are take on this system:

POSTGIS=”2.0.1SVN r9637″ GEOS=”3.4.0dev-CAPI-1.8.0″ PROJ=”Rel. 4.8.0, 6 March 2012″ GDAL=”GDAL 1.9.0, released 2011/12/29″ LIBXML=”2.7.6″ LIBJSON=”UNKNOWN” TOPOLOGY RASTER

PostgreSQL 8.4.10 on x86_64-pc-linux-gnu, compiled by GCC gcc-4.4.real (Ubuntu 4.4.3-4ubuntu5) 4.4.3, 64-bit

CUSTOM OPTIONS:
shared_buffers = 128MB      (default is 24MB)
temp_buffers = 32MB         (default is 8MB)
work_mem = 8MB              (default is 1MB)
maintenance_work_mem = 32MB (default is 16MB)
max_stack_depth = 4MB       (default is 2MB)
checkpoint_segments = 24    (default is 3)

CPU: Intel(R) Core(TM)2 Duo CPU P9500 @ 2.53GHz (2 x 5054.34 bogomips)

RAM: 4GB

Considerations

The procedure described in this post is also valid for LINESTRING and MULTILINESTRING layers, using the exactly same code.

You could reuse the topology to produce multiple resolution levels w/out incurring again in the construction cost (and with a reduced input complexity at each level).

The simplification step doesn’t use TopoGeometry objects at all so you could choose to perform topology construction and attribute assignment in a different way.

Running the SimplifyEdgeGeom function again might give you more simplification because edges which may have intersected to the simplified version of an edge may not be intersecting anymore after their own simplification. The function can be changed to behave differently on exception to improve performance or quality.

PostGIS 2.0.0 released

Wed, 04 Apr 2012 09:46:16 +0000

The long-awaited full featured PostGIS 2.0.0 is finally out. Coupled with GEOS 3.3.3 (released a few days before) and GDAL-1.9.0, it brings you the best spatial database system in town, complete with raster analysis and topology modeling support.

Complete announcement, with list of changes, here.

It’s been an great pleasure to work with the rest of the team on getting this release ready for shipping, drawing a line after over two years of hard work on new features. I’m particularly proud of the persistent topology support, which kept me busy for the most part of 2011, and the raster support of which I wrote the foundations two years before.

I’m also happy to see an healthily growing community around PostGIS: we’ve had two successful pledges, a growing list of contributors and more corporate sponsors.

A special thank goes to Vizzuality for investing in a full-time PostGIS hacker. Their PostGIS-in-the-cloud solution (cartodb.com) is likely the first one putting PostGIS 2.0.0 in production.

Gnash 0.8.10 released

Tue, 07 Feb 2012 08:45:41 +0000

After one year of gestation, Gnash 0.8.10 is finally out.

It fixes many compatibility issues (fixing from Google Dict to Camtasia and Captivate outputs, to a while category of interactive games), enhances user experience (popups on limits hit, gnome thumbnailer, QT4 mousewheel support), implements more of the SWF8 specs (BitmapData perlin noise), introduces new accelerated renderer (OpenVG) and better framebuffer GUI (touchscreen aware).

This is the first Gnash release after Adobe announcement of giving up Flash for HTML5 in the mobile market. Battery life is king for mobile devices, so being able to enable compile-time optimizations both for general or specific SWF interpretation makes Gnash a solid alternative.

A walk on the wild side

Sat, 28 Jan 2012 00:03:20 +0000

I’ve been spending the last few days profiling and optimizing the new simple-to-topological converter you will find in PostGIS 2.0.0 thanks to a community effort.

The most expensive operation was found to be the ST_AddEdgeModFace function, which adds an edge and checks if such edge creates a new face.

Face-splitting detection was implemented using a brute force approach consisting in invoking the GEOS polygonizer and then checking if any polygon created contained the newly added edge in its boundary. It can get pretty wild when you add an edge in the universe face, and thus end up feeding the polygonizer with thousands of edges.

So I started thinking about two fields we make great effort to maintain and were (so far) unused: next_left_edge and next_right_edge. Each edge has these two pointers which may be used to walk clockwise along its left or right face.

By walking on the edge side you can know a lot about the edge you add !

First of all if you get to the other side of the edge you know for sure you didn’t split any face (no ring was created). Second, you can easily get the identifiers of all the edges you’ll need to update for setting their new left_face and right_face attributes. Finally by computing ring winding you can know whether each side of the edge is forming a fill or an hole.

I was afraid that such approach could have been slower than the brute force one, mainly due to more database querying and IO. But the dataset I was using for a testcase (Italian municipalities by ISTAT: 8094 multipolygons with an average of 560 vertices each) showed a speedup of up to 10x !

The testsuite was an invaluable tool for correctness checking. I actually spent a fair amount of time enhancing it further with new corner cases (new algorithm, new corners!).

I’m still checking correctness of the resulting topology for the ISTAT case, making sure that the improvements didn’t introduce any regression. Hopefully next week the code will be committed upstream for broader delectation. Stay tuned, prepare your input for conversion and start playing and timing the current routines !

GEOS 3.3.2 released

Fri, 06 Jan 2012 21:12:04 +0000

The second bugfix release in the 3.3 branch of GEOS was released today.

This is the version required by the topology support shipped with the upcoming PostGIS 2.0 release.

Everyone is recommended to upgrade. Changes can be read here, package can be downloaded here.

Topology cleaning with PostGIS

Mon, 21 Nov 2011 17:51:37 +0000

An early tester of the new PostGIS Topology submitted an interesting dataset which kept me busy for a couple of weeks fixing a bunch of bugs related to numerical stability/robustness.

Finally, the ST_CreateTopoGeo function succeeded and imported the dataset as a proper topological schema. Here’s what it looks like:

Edges of the built topology

At a first glance it doesn’t seem to be particularly problematic. Here’s the composition summary:

=# select topologysummary('small_sample_topo');
                    topologysummary
--------------------------------------------------------
 Topology small_sample_topo (2042), SRID 0, precision 0
 83 nodes, 156 edges, 74 faces, 0 topogeoms in 0 layers

But the devil hides at high zoom levels. Where to zoom ? What are we looking for ?

We are guaranteed none of the constructed edges cross so the only leftover problem we might encounter is very small faces constructed wherever the original input had small overlaps or underlaps (gaps). We can have a visual signal of those faces by creating a view showing faces with an area below a given threshold. Let’s do that:

CREATE VIEW small_sample_topo.small_areas AS
SELECT face_id, st_getfacegeometry('small_sample_topo', face_id)
FROM small_sample_topo.face
WHERE face_id > 0
AND st_area(st_getfacegeometry('small_sample_topo', face_id)) < 0.1;

That query would let us see where to find faces with area < 0.1 units. And here’s qgis showing it:

Areas smaller than 0.1 square units

Now we know where to zoom, and also the ID of the offending faces.

Let’s zoom in and show some labels:

Detail of small area

You can now see that face 59 is bound by (among others) edges 130 and 129. Just get rid of one of them to assign the area to an adjacent face.

We drop edge 130 using ST_RemEdgeModFace, assigning the area to face 52. Here’s the result:

Area after cleanup

Cleaning further would require removing further edges and thus getting rid of all the small faces. There’s a lot of room for automating such processes. The good new is you can now build your own automation around your specific use cases while still using robust and standard foundations.

It is to be noted that the whole process I described here only involved the geometrical/topological level and didn’t affect at all the semantic/feature level. If we had TopoGeometry objects defined by the faces we’d also know which small faces were part of overlaps or underlaps and could then act consequently by adding or removing faces to the definition of the appropriate TopoGeometry object. Such step would have been required for the overlap situations as the ST__RemEdgeModFace_ function doesn’t let you change the shape of a defined TopoGeometry.

Unfortunately the semantic level is lost when using the ISO functions, as the whole ISO topology model doesn’t deal with features at all. This is why I think PostGIS would benefit from having a function that converts your simple features into topologically-defined features by adding any missing primitive to a topology schema and constructing the feature for you. Such function, is only waiting for sponsorship to become a reality of PostGIS 2.0. If you like what we’re building for your data integrity, please consider supporting the effort!

Ming 0.4.4 released

Wed, 26 Oct 2011 09:21:45 +0000

I’m happy to announce that release 0.4.4 of the Ming library is out.

Ming is an SWF output library with binding for C, C++, PHP, Perl, Python and more.

It’s stable, alive and waiting for you at his new [github][1] location.

Don’t let Flash get in your way, do your part for an open web!

Changes in this release:

Generally improve swftoscript and decompiler
Change makefdb to name output files by font ID, to play nicer with swftoscript.
Add support for ‘class A extends B’ syntax in actioncompiler
Fix bug in ‘makeswf’ failing to catch some compile errors (bugzilla #94) and being too silent in swf embedding errors
Fix bug in action compiler dealing with class methods (bugzilla #94)
Add support for libpng > 1.4 (bugzilla #96)
Add font kernings support (bugzilla #95)
Add button characters export capabilities
Add support for ‘swfAction <code>’ syntax in asm blocks

PostGIS topology ISO SQL/MM complete

Fri, 14 Oct 2011 13:39:10 +0000

PostGIS implementation of the ISO SQL/MM Topology-Geometry model is finally complete.

The SQL/MM model is just a portion of the whole topology support, but an important one, including schema definition and functions to create and populate the schema with primitive components (nodes, edges, faces).

In addition to the base model, PostGIS adds a TopoGeometry data type for use wherever you would normally use a Geometry type, except the former will be defined by references to primitive, shared, topological components.

The work on SQL/MM topology for PostGIS 2.0 was mainly funded by the Land Management department of the Tuscany Region as part of a larger project aimed at enhancing effectiveness of free software tools already in use within the institution for the management of the topographic database (chapeau!).

But more work is needed to make it easier to convert your layers to topological layers, your Geometries to TopoGeometries. Because I know you want to generalize your vectors in a topologically consistent way, don’t you ? And you want to edit the boundaries just once. And do you want to waste all that space for your triangles ?

Feature freeze for 2.0 will be by the end of November. Drop a note ASAP if you’d like to help with getting the enhanced TopoGeometry constructor shipped with it.

GEOS 3.3.1 released

Tue, 27 Sep 2011 16:44:22 +0000

The first bugfix release in the 3.3 branch of GEOS was released today.

Anyone running 3.3.0 is recommended to upgrade. Changes can be read here, package can be downloaded here.

By the way… did you try the PHP binding yet ? Configure with --enable-php to take a look !

Hack fever 2 : super mario sound

Wed, 07 Sep 2011 13:52:13 +0000

I was supposed to leave for holidays on August 1st, but a summer fever held me back.

As it became traditional, I took the chance to do some Gnash hacking, approaching a bug that made playing Super Mario Bros unpleasant. The mario sound didn’t stop on game over thus overlapping with the new sound started for the new game. Annoying.

After some research the help came from Jan Flanders, an usual attender of the #gnash IRC channel. He pointed out that calling .stop() against an unattached Sound object should stop all playing event sounds. It took a couple of days from there to an automated testcase and a fix.

You can now Enjoy Your Mushrooms !

Getting just the tip of a remote git branch

Tue, 07 Jun 2011 16:50:11 +0000

As projects move their code under git control, people get frustrated about being unable to do most basic operations they are used to perform with SVN or CVS. That’s a fact, so let’s see if I can relief some pain by sharing what I know or learn as I crawl the learning curve myself.

Yesterday I’ve met with Markus Neteler and he was complaining about being unable to checkout the release branch of QuantumGIS without filling up his laptop hard drive. He got me curious, so here are some numbers and a recipe.

An SVN checkout as of April 30 is 281 Mb in size, 1 of which being the .svn directory.

A full git clone at time of writing (June 7) is 330 Mb in size, 200 of which being the .git database and 130 being the working copy (the “checkout”).

A full git clone contains all the data available in the original repository. Once you get the clone, you have all the branches and all the history. ****no need for any more bandwidth.

But Markus was only interested in a single branch, not the whole set, and he wanted no history either. So he could cloned just the objects referenced by the commit known as the release-1_7_0 branch and no further parents (back history). Here’s how you do:

 git clone --depth 1 --branch release-1_7_0 \
           git://github.com/qgis/Quantum-GIS.git

The resulting shallow repository (the .git directory) is 110 MB in size. Add 133 MB of working directory (yes, release-1_7_0 is 3 MB bigger than master) for a total of 243 MB disk space used.

NOTES:

A shallow repository (one with short history) cannot be further cloned, but here are no problems pulling updates from the origin nor producing patches or pushing changes.
If you don’t know in advance the name of the branch you can query it from the remote repository using git ls-remote
Every git command has a manual page in the form: git-command (ie: man git-ls-remote)

Happy learning !

GEOS 3.3.0

Tue, 31 May 2011 07:57:22 +0000

GEOS 3.3.0 is out: http://download.osgeo.org/geos/geos-3.3.0.tar.bz2

This release introduces a fair amount of new C-API interfaces and a brand new PHP binding. Full details in the NEWS file: http://trac.osgeo.org/geos/browser/tags/3.3.0/NEWS

As with any release since 3.0.0 there is complete binary compatibility with clients linked against the C-API. These include, but are not limited to, PostGIS. For a list of known clients: http://trac.osgeo.org/geos/wiki/Applications (add yours, if not already listed!)

GEOS is a C++ port of the JTS Topology Suite. This release targets version 1.12 of the library, but doesn’t reach full feature parity yet. Missing JTS functionalities:

Densifier class
Geometric similarity detection package (HausdorffSimilarityMeasure, AreaSimilarityMeasure)
MinimumDiameter.getMiminumRectangle()
Triangulation API
VoronoiDiagramBuilder
createSquircle and createSuperCircle in GeometricShapeFactory
MinimumClearance class
nearestNeighbours method to STRtree
RandomPointsBuilder / RandomPointsInGridBuilder
KochSnowflakeBuilder
SierpinksiCarpetBuilder

If you’d like to sponsor development of any of the above items (or others) for next feature release of GEOS (3.4.0) please drop me a note.

Software libero e tecnologie partecipative

Mon, 30 May 2011 09:06:08 +0000

Lunedi’ 6 Giugno 2011 alle 20:30 presso la sede dell’ associazione Cubibi’ di Ispra si terra’ un’ incontro aperto sul software libero e le tecnologie partecipative.

L’ incontro e’ rivolto a tutti coloro che coltivano la propria curiosita’ come una risorsa e vogliono scoprire qualcosa in piu’ sugli strumenti elettronici che mediano sempre piu’ le nostre attivita’ personali e sociali estendendone o limitandone le possibilita’, aiutando od ostacolando la nostra capacita’ di scelta.

UPDATE: qui il materiale per stampare il volantino.

PostGIS / GEOS / MapServer with git

Wed, 27 Apr 2011 21:07:53 +0000

I’ve setup git mirrors of PostGIS, GEOS and MapServer SVN repositories updated hourly. You can clone the git repositories and re-attach to the SVN ones with this simple script (untested):

for repo in postgis geos mapserver; do
  git clone git://github.com/strk/${repo}.git
  cd ${repo}
  git svn init http://svn.osgeo.org/${repo}/trunk
  git update-ref refs/remotes/git-svn refs/heads/master
  git svn fetch
  cd -
done

After that you can run git svn rebase to get changes from SVN or git pull to get changes from GIT (may be one hour late)

Happy hacking !

Why Political Liberty Depends on Software Freedom More Than Ever

Fri, 11 Mar 2011 10:03:10 +0000

I’ve spent a few days adding Italian subtitles to a video recording of the speech given by Eben Moglen at FOSDEM 2011 about privacy, freedom and net neutrality.

I would have liked to embed the video right here, but these advanced friendly systems make everything so… ehm… difficult !

Well, take a look at the simple version of it 🙂

Hack Fever

Fri, 24 Sep 2010 09:33:42 +0000

Got seasonal flu this week, forcing me home… Nothing better for some Gnash hacking !

I had this itch to scratch for degradation of experience in playing the wonderful Winterbells game.

It started in June when I realized that Gnash-0.8.8 could not start the game properly and was suspiciously slower than 0.8.7. In late July I put playback under profile and found out that the performance penalty was introduced by a compatibility fix (property name case in SWF up to 6).

So, with high temperature and kid at school, I resolved to do something about it: first I fixed the startup problem, then I created an objecturi git branch to proof-test caching of lowercase property keys as a mechanism to improve lookups.

Now it’s friday and flu is over, so I need to clean up and put the toys away.

Luckly, we are talking about free (as in freedom) software, so it’s easy to find friends caring about the toys and happy to play togheter! So I had a chat with Ben resulting in a proper plan, and he agreed to go on playing some more and cleanup afterwards.

Ain’t Free software wonderful ?

Feed the bats

Thu, 22 Apr 2010 20:28:14 +0000

My bats are gone with the spring, leaving behind what looks like delicious food!

These insects came out from who knows where and filled up the walls of our house.

They do look like Clothing Moths and probably are, but contrary to what wikipedia says they don’t seem to prefer low light as we often see them around light bulbs.

This invasion is mining our will to remain in the house, which is a pity as there’s a wonderful garden out here and flowers are popping out on a daily basis :/

UNICODE from OSM to PGSQL (part 2)

Mon, 05 Apr 2010 11:08:09 +0000

There is no problem importing OSM data into PostgresSQL / PostGIS.

In part one of the article we’ve seen Geofabrik’s shapefiles having a text data truncation problem, but using osm2pgsql everything gets into an UTF-8 database without a failure.

It’s as simple as:

$ osm2pgsql -l -c -S default.style africa.osm.bz2 -d osm

The -l switch aks for keeping lat/long projection, -c requests creation of the schema, -d specifies the database to use. The default.style file is a configuration specifying what to import and how; I used the default for the sake of this test.

Resulting ralations:

Schema |        Name        |   Type
--------+--------------------+----------
 public | planet_osm_line    | table
 public | planet_osm_point   | table
 public | planet_osm_polygon | table
 public | planet_osm_roads   | table

And the way we’ve been using for testing has all characters:

Let’s see it in Quantum GIS, compared with the one coming from the corrupted shapefile (which I’ve imported into postgis after hacking shp2pgsql to discard incomplete multibytes):

The difference you may notice seems to be due to left-to-right vs. right-to-left orientation of the text. My terminal seems to ignore orientation, qgis doesn’t.

Now, time to see if a shapefile will be able to bear all that UNICODE. Let’s not do anything fancy, just dump the roads table using pgsql2shp:

$ pgsql2shp osm planet_osm_roads

Pretty fast (slightly above 1 second system time, 8 secs real time). And here’s the generated shapefile dataset:

72979148 planet_osm_roads.shp
69555432 planet_osm_roads.dbf
  516268 planet_osm_roads.shx
     257 planet_osm_roads.prj

Do they have the full multibyte strings now ? Sure, shp2pgsql doesn’t complain anymore, and you can safely import into postgis again completing the round-trip. Only you have to specify input encoding UTF-8 as the new default encoding, as I pointed out in previous post, is that unmentionable one… So:

$ shp2pgsql -W UTF-8 planet_osm_roads planet_osm_roads_roundtrip | psql osm
...
$ psql osm -c 'select name from planet_osm_roads_roundtrip where osm_id = 4005333';
 Avenue des Nations Unies - شارع الأمم المتحدة

Also, we can open the shapefile itself with qgis and see how it looks:

Green is the pgsql2shp-exported shapefile, red is osm2pgsql-imported planet osm, black is geofabrik-imported shapefile.

All clean and easy 🙂

Further excercises would include tweaking the osm2pgsql style file and generally the import process to better select data of interest, properly clean geometry invalidities and taking care of incremental updates of the data.

Good luck and happy hacking !

UNICODE from OSM to PGSQL

Sun, 04 Apr 2010 22:09:30 +0000

This week I’ve been presented with a problem importing OpenStreetMap data of Africa from GeoFabrik’s shapefile export into a PostgreSQL / PostGIS database.

The problem consisted in a loss of information during the transport, resulting in wrongly encoded strings (road names) ending up in the db. This was during a feasibility study. So, is that feasible ? Let’s take a look.

I downloaded the shapefiles and tried to import the roads one using shp2pgsql with no options, and here’s the result:

Unable to convert field value "Place Othman Ibn Affane ساحة عثمان اِ" to UTF-8:
iconv reports "Invalid or incomplete multibyte or wide character"

Why is shp2pgsql trying to convert, and from which encoding? When I left it, the default was to perform no conversion unless -W was given…

Well, it turns out the default is now to convert from WINDOWS-1252 encoding (why?) and there no way to request no encoding at all (why?!).

So I patched the loader to give more informations about the encoding process and specified UTF-8 as source encoding_. Here’s the result_:

Unable to convert field value "Avenue des Nations Unies - شارع الأمم " from UTF-8 to UTF-8:
iconv reports "Invalid argument"

So it’s official: the dbf file contains invalid data. The confusing error message (Invalid argument) means the multibyte sequence is incomplete rather than invalid (EINVAL errno).

Adding more debugging code I can see that many many rows have values that look truncated, all ending with a single byte being either <D8> or <D9>.

OpenOffice confirms the malformation (wanted an independent opinion on that just in case it was shapelib doing the truncation):

Querying openstreetmap for way 4005333 shows the full string, and the full string is also present in the .osm file downloaded from geofabrik:

” width=”549″ height=”23″>

So the problem is only with the shapefile, not the OSM data itself, nor with postgis.

Surely postgis loader could be tweaked to allow for a tolerance, in case anyone wants to import the truncated data anyway. In this specific case discarding the final partial multibyte string might be the best you can do as it’s a case of truncation as any other, only being multibyte it gives more problems than single-byte encoding.

Timely enough someone submitted a patch aimed at exactly this kind of tolerance handling. I’m going to see how well that’ll cope with this case.

But bottom line is we do want the good data, so this problem is not solved until the data will be in the database, stepping by shapefiles (if possible) or directly.

Next stop: osm2pgsql -> go there

Meet the bats

Sun, 14 Feb 2010 10:34:31 +0000

Since I moved in a new house I got to know bats.

Could be due to the fact the house was inhabitated for some time. Thing is, every now and then a bat or two come out from who-knows-where and start flying around the sitting room.

First time we met it was at late night. Diana woke me up scared. “What’s that thing flying around?!” It took a lot of time and cold before we succeeded at sending it out the window. I was surprised how few intentions he had to get out. Wintertime. Very cold. We surely didn’t keep the windows open during the day. Where did that bat come from ?

We left the house for christmas/new-year and when we came back two bats were flying in the sitting room. This time we’ve been nicer. Also the kid was happy to see them (she’s always wanted pets). This time, they also sometime stopped flying and took some rest against a wall. Finally they disappeared (again after damn cold entering the house).

Then it happened again after leaving the house for jusr a few hours. This time I didn’t handle to let the bat out, but he disappeared anyway. God knows where. We tried to find him looking in every place of the house with no luck. “Well, we’ll see tomorrow”. Gone sleeping. The day after, while I was alone in the house, silently tapping on the computer keyboard, light wings noise I’ve heard, then not much later the bat showed up again flying in circle. This time I took the chance to take some pictures.

The bat didn’t fly away when I tried to pick him with my hands (and a blanket) after it took a rest on the window curtain. So I took the chance to take more pictures, then took him out the window and he flied from my hand.

I start to suspect there’s actually a colony in the house which we’ll need to learn co-habit with. Hopefully animal experts will have some advice for me. Anyone ?

About this blog

Mon, 01 Jan 0001 00:00:00 +0000

This is my personal blog, converted from Wordpress to Hugo around in May 2020, following a flood of spam and scarcity of RAM on my server.

Comments might be harder to make now, but you can always send me an email and I’ll see how to add them manually.

Beside this blog, which is managed via Hugo, the rest of my website is fully done manually. Until I’ll migrate it all, you can find it from the root path on this domain.