Communauté

Où en est le BIM pour les Infrastructures ?

Geospatial France - ven, 02/02/2018 - 17:27
Dodge Data & Analytics a mis récemment à disposition un rapport sur “la valeur du BIM pour les Infrastructures 2017”. Vincent Fredon en parlait aussi sur Civil Made in France. Les pays couverts...

{Cliquez sur le titre pour lire la suite...}

Vues panoramiques dans AutoCAD, Map 3D et Civil 3D-Webinaire Brockwell et Cyclomedia le 8 février 2018

Geospatial France - mer, 01/24/2018 - 09:12
Le 8 février 2018 à 16h, Brockwell et Cyclomedia organisent un webinaire sur l’utilisation des vues panoramiques 360 degrés de Cyclomedia sous Autocad (aussi pour Map 3D et Civil 3D). Brockwell est...

{Cliquez sur le titre pour lire la suite...}

My GIS performance impact of Meltdown and Spectre patches

Geomatips - mar, 01/09/2018 - 19:34
When I saw performance regression few days ago of Meltdown and Spectre patch from cpu conception problem, I would like to perform some basic test to understand if it could slow down my process.

For those who don't know, here : a postgres article about performance regression : https://www.postgresql.org/message-id/20180102222354.qikjmf7dvnjgbkxe@alap3.anarazel.de 
Since some users finished to make benchmark with Meltdown and Spectre patch, I will not test it on my computer.

How to test  ?
First, I tested it on Archlinux based distribution named Manjaro Linux.


To know if patch is applied to my kernel, I just run this command :

$ cat /proc/cpuinfo | grep cpu_insecureWhich reports this :

bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecureWhen showing cpu_insecure, you know that KPTI patch is enable.

Then there is a specific C code to test your cpu : 

    #include
    #include
    #include
    #include
    // gcc -Wall -O3 -o getpid getpid.c
    int main(void)
    {
    long value;
    clock_t begin = clock();  
    for (int i = 0; i < 100000000; i++)
     value = syscall(SYS_getpid);
    clock_t end = clock();
    double time_spent = (double)(end - begin) / CLOCKS_PER_SEC;
    printf("%d\n", (int) value);
    printf("%f\n", time_spent);
    return 0;
    }


Write these lines to pti.c file, build it with :

gcc -Wall -O3 -o pti pti.c
and finally run it with
./ptiFor my cpu aka Intel i7 4790k, switching to patched kernel, my computer slow down a lot from this generic test. When upgrading to pacthed kernel, you can still disable patch with kernel grub option pti=off. Next tests will be shown with pti=on and pti=off

With pti=off and generic test, my results could be shown below :


1211
3.571401 seconds With pti=on and generic, we could watch a several regression :
1208
13.998673 seconds Now to know when rebooting my system, I know if patch is apply or not and before making tests, I will build and run this C code.
Simple GIS test : 
From years now and unfortunately, I fight with ECW files. I have some big ecw which I found far away better in performance in comparison to multiple files like 36GB.
But for this test I will use a moderate ECW raster which is an orthophotography of Mauritius Island with 15cm resolution pixel and size of this file is 15 GB and cover 56 x 67 kilometers. Sample overview :

For those are curious here is in detail :

$ gdalinfo ALL.ecw
Driver: ECW/ERDAS Compressed Wavelets (SDK 3.x)
Files: ALL.ecw
       ALL.ecw.aux.xml
Size is 372000, 450000
Coordinate System is:
PROJCS["SUTM40",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            TOWGS84[0,0,0,0,0,0,0],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9108"]],
        AXIS["Lat",NORTH],
        AXIS["Long",EAST],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",57],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",10000000],
    UNIT["Meter",1]]
Origin = (531000.000000000000000,7796700.000000000000000)
Pixel Size = (0.149999999999993,-0.149999999999993)
Metadata:
  COLORSPACE=RGB
  COMPRESSION_RATE_TARGET=4
  VERSION=2
Corner Coordinates:
Upper Left  (  531000.000, 7796700.000) ( 57d17'46.37"E, 19d55'32.96"S)
Lower Left  (  531000.000, 7729200.000) ( 57d17'50.53"E, 20d32' 8.78"S)
Upper Right (  586800.000, 7796700.000) ( 57d49'45.72"E, 19d55'26.88"S)
Lower Right (  586800.000, 7729200.000) ( 57d49'57.39"E, 20d32' 2.50"S)
Center      (  558900.000, 7762950.000) ( 57d33'49.99"E, 20d13'48.53"S)
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=59.587, StdDev=56.411
  Overviews: 186000x225000, 93000x112500, 46500x56250, 23250x28125, 11625x14062, 5812x7031, 2906x3515, 1453x1757, 726x878, 363x439, 181x219
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=59.586901452342
    STATISTICS_MINIMUM=0
    STATISTICS_STDDEV=56.410809243179
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=57.965, StdDev=53.573
  Overviews: 186000x225000, 93000x112500, 46500x56250, 23250x28125, 11625x14062, 5812x7031, 2906x3515, 1453x1757, 726x878, 363x439, 181x219
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=57.964855468501
    STATISTICS_MINIMUM=0
    STATISTICS_STDDEV=53.572679192972
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=47.609, StdDev=46.083
  Overviews: 186000x225000, 93000x112500, 46500x56250, 23250x28125, 11625x14062, 5812x7031, 2906x3515, 1453x1757, 726x878, 363x439, 181x219
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=47.608960074074
    STATISTICS_MINIMUM=0
    STATISTICS_STDDEV=46.082615455709


Idea is to create a pretty well compress geotiff (and a good way to switch from ECW to geotiff since Paul Ramsey make a great tutorial about it and it's now embeded in Qgis raster menu ! ) to understand if these kernel patches have influences over a standard GIS tranformation.
I use gdal_translate with time command in a bash terminal to benchmark it. Command line is like that :

$ time gdal_translate -co BIGTIFF=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES -co NUM_THREADS=8 in.ecw out.tif With pti=off I've got this result : 


Input file size is 372000, 450000
0...10...20...30...40...50...60...70...80...90...100 - done.

real 113m36,193s
user 106m22,559s
sys 0m34,030s And with pti=on this :
Input file size is 372000, 450000
0...10...20...30...40...50...60...70...80...90...100 - done.

real 117m4,998s
user 107m28,128s
sys 0m44,773s
Results shiow +/- no difference which demonstrate there is no impact regression on a Gdal GIS usage. Time for this process is big due to usage of JPEG compression which is only single core in gdal. Test is perform on external usb 3.0 2.5" hard drive.I made some others tests with gdalwrap and multithreading option and also there is no significant differences ...CPU / GPU test with blender and generic blenchmark : 
Finally, I use my computer to render 3D files sometimes and I would like to test CPU and GPU in this test. May be not related but cpu manage (with a small amount) gpu during gpu computation.

With pti on or off I had +/- same results : 45/47s with dual GPU and 2min10 / 2min11




So applying and using patch is recommanded since they publish how to use this leaks but we clearly show that they want to sell new platforms with a great new performances (in fact same as before without KPTI patches).
Happy proofing and selling thing but in my case it doesn't matter except with my postgres / postgis database. Does postgres tests made by others with pgbench really perform impact of GIS usage ?!

Catégories: Technique

Additions to the MapFish Protocol

Eric Lemoine - sam, 04/18/2009 - 23:55

We recently added new stuff to the MapFish Protocol.

As a refresher, let’s first take a look at what the MapFish Protocol had before the new additions.

(Note that you’d need the JSONovich FireFox extension to see the output of the examples given below in your web browser.)

Geographic query params

  • box={x1},{y1},{x2},{y2}: the features within the specified bounding box
  • geometry={geojson_string}: the features within the specified geometry
  • lon={lon}&lat={lat}&tolerance={tol}: the features within the specified tolerance of the specified lon/lat

Examples:

Limiting and Sorting

  • limit={num}: the maximum number of features returned
  • offset={num}: the number of features to skip
  • order_by={field_name}: the name of the field to use to order the features
  • dir=ASC|DESC: the ordering direction

Examples:

The new params

  • no_geom=true|false: so that the returned feature has no geometry (“geometry”: null)
  • attrs={field1}[,{field2},...]: to restrict the list of properties returned in the features
  • queryable={field1}[,{field2},...]: the names of the feature fields that can be queried
  • {field}__{query_op}={value}: filter expression, field must be in the list of fields specified by queryable, query_op is one of “eq”, “ne”, “lt, “le”, “gt”, “ge”, “like”, “ilike”

And now an example combining all the new parameters:

The above query returns a GeoJSON representation of the summits whose names include “col” and whose elevations are greater than or equal to 3500. The returned features have no geometry and their attributes include “name” and “elevation” only.

Not including the geometry in the features makes the parsing in the browser much faster, so for cases where the geometries aren’t needed this is a big win.

Credits for the “queryable={field}&{field}__{query_op}={value}” syntax goes to FeatureServer!


Catégories: Technique

Secure TileCache With Pylons and Repoze

Eric Lemoine - dim, 02/15/2009 - 19:14

This post shows how one can secure TileCache with Pylons and Repoze.

In a Pylons application one can run a WSGI application from within a controller action. Here is a simple example:

class MainController(BaseController) def action(self, environ, start_response): return wsgiApp(environ, start_response)

TileCache is commonly run from within mod_python. TileCache can also be run as a WSGI application, therefore it can be run from within the controller action of a Pylons application. Here’s how:

from TileCache.Service import wsgiApp class MainController(BaseController) def tilecache(self, environ, start_response): return wsgiApp(environ, start_response)

Pretty cool… But it gets really interesting when repoze.what is added to the picture. For those who don’t know repoze.what, repoze.what is an authorization framework for WSGI applications. repoze.what now provides a Pylons plugin, making it easy to protect controllers and controller actions in a Pylons application. Here’s how our tilecache action can be protected:

from TileCache.Service import wsgiApp from repoze.what.predicates import has_permission from repoze.what.plugins.pylonshq import ActionProtector class MainController(BaseController) @ActionProtector(has_permission('tilecache')) def tilecache(self, environ, start_response): return wsgiApp(environ, start_response)

With the above, anyone who tries to access /tilecache will have to be granted the tilecache permission. Otherwise, authorization will be denied.

TileCache is secured!

People often want finer-grained authorization, like give certain users access to certain layers. With Pylons’ routing system this can be easily and elegantly achieved using repoze.what, I will show that in a later post.


Catégories: Technique
S'abonner à OSGeo-fr agrégateur