Présentation du FME World Tour du 13 avril 2017 à Quebec
•Download as PPTX, PDF•
1 like•188 views
Présentation de l'événement FME World Tour 2017 qui a eu lieu le 13 avril 2017 à Québec. Découvrez les nouveautés de FME 2017 et FME Server 2017. Voyez les trucs et astuces pour optimiser la performance de vos workbench, une solution pour comparer des workspaces ensemble, un portail de chargement et téléchargement de données avec FME Server ainsi que des outils de validation et correction topologique.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Présentation du FME World Tour du 13 avril 2017 à Quebec
Similar to Présentation du FME World Tour du 13 avril 2017 à Quebec (20)
Recently uploaded
Recently uploaded (20)
Présentation du FME World Tour du 13 avril 2017 à Quebec
- 1. Bienvenue Québec – 13 avril
- 2. Programme (avant-midi) 8h45 Mot de bienvenue 9h00 Survol des nouveautés de FME 2017 9h15 Trucs et astuces pour optimiser la performance de vos workbench dans FME 2017 10h00 Pause-café 10h30 Time Machines and Attribute Alchemy 11h00 Capsules et astuces de Consortech • Comparateur de Workspaces • Chargement et téléchargement de données avec FME Server • Validation et correction topologique 11h45 Dîner
- 3. Programme (après-midi) 13h00 Quoi de neuf dans FME Server 2017 • Automatisation des données corporatives (Enterprise Automation) 13h30 Actions automatisées en temps reel avec FME Server 14h15 Jeu interactif 14h45 Conclusion
- 5. Fiers partenaires depuis 1999
- 6. Mission Vous permettre d’accéder à une information de qualité afin de prendre des décisions éclairées pouvant bénéficier à l’ensemble de la collectivité.
- 7. Qui aidons- nous? Organisations qui gèrent du territoire et/ou des infrastructures : Villes, MRC et régions Gouvernements Utilités publiques
- 8. Intégration de données • ETL • Connectivité intelligente • Valorisation des données Notre spécialité, Les données Expertise
- 9. Services-conseils visant à : • Extraire • Transformer • Organiser • Automatiser • Intégrer les données efficacement peu importe l’architecture en Expertise
- 11. Vos défis Productivité : • Automatiser les tâches redondantes • Accélérer le traitement des données provenant de sources multiples Qualité : • Assurer la conformité des données selon des normes
- 12. Vos défis Coûts : • Réduire les coûts liés à l’extraction, la validation, la publication et l’intégration de vos données organisationnelles
- 13. Offre de services Trousse de démarrage pour FME Chargement/ téléchargement de données Automatisation des données corporatives
- 16. Bonne journée!
- 17. Survol des nouveautés de FME 2017 Par Annabelle Glass, Spécialiste FME chez Safe Software
- 18. Why we do what we do: Data should be free to use wherever, whenever and however it’s needed. Information should flow between systems, applications, and formats. You should spend your time making decisions, not fighting with data.
- 19. “Any sufficiently advanced technology is indistinguishable from magic” - Arthur C Clarke
- 20. FME 2017 ✔ Connect more systems ✔ Perform powerful transformations ✔ Create workflows quickly and easily ✔ Inspect any data, any time ✔ Automate anything
- 21. Who has the Magic Ticket? One Lizard Wizard USB drive holds a free pass to the FME International User Conference 2017 May 24-26 in Vancouver, BC + 5 Other Chances to Win FME Prize Packs fme.ly/magicticket
- 23. Get all of today’s talks at safe.com/presentations
- 24. Let’s go to magic school.
- 25. Goal: Turn raw satellite imagery into a useful 3D model.
- 26. What’s New in FME 2017.0
- 27. New formats • Autodesk FBX (R/W) • BIM Collaboration Format (BCF) (R/W) • Cesium 3D Tiles (W) • glTF (W) • HTML Table (R) • IBM dashDB (R/W) • MapBox MBTiles (R/W) • MapInfo Extended TAB (R) • OGC Geopackage Raster Tiles (R/W) • OpenStreetMap PBF (R) • Planet (R) • QlikMaps (R/W) • Sentinel-2 on AWS (R) • STL (R/W) • New background map: Japan GSI Maps • New on OS X: Tableau (W), Collada (R/W) • New on Linux: Tableau (W)
- 28. FME 2017 ✔ Connect more systems ✔ Perform powerful transformations ✔ Create workflows quickly and easily ✔ Inspect any data, any time ✔ Automate anything
- 30. New transformers • AutodeskA360Connector • BoxConnector • DatabaseJoiner (renamed from Joiner) • DropboxConnector (renamed from DropboxCaller) • FMEServerResourceConnector • Geocoder • GoogleDriveConnector • GridInQuestIIReprojector • HTMLExtractor • LineBuilder (renamed from PointConnector) • LineExtender (renamed from Extender) • OneDriveConnector • PointCloudSimplifier • PointCloudSurfaceBuilder
- 36. Put Your Date/Times in a Bottle (or at least a transformer)
- 37. FME 2017 ✔ Connect more systems ✔ Perform powerful transformations ✔ Create workflows quickly and easily ✔ Inspect any data, any time ✔ Automate anything
- 39. Add formats with Quick Add.
- 41. Transformer parameters are a dockable pane.
- 44. “Open Containing Folder” and “Inspect” transformers that use an external file.
- 45. Choose how to handle rejected features.
- 46. Copy and paste parameters.
- 48. WEB SERVICES
- 52. WEB FILESYSTEMS
- 53. Browse web data like a local filesystem.
- 54. Manage data stored remotely.
- 55. FME 2017 ✔ Connect more systems ✔ Perform powerful transformations ✔ Create workflows quickly and easily ✔ Inspect any data, any time ✔ Automate anything
- 56. Reveal the Prince or Princess in your data Frog (no kissing necessary!)
- 58. FME Data Inspector: Table view
- 59. FME 2017 ✔ Connect more systems ✔ Perform powerful transformations ✔ Create workflows quickly and easily ✔ Inspect any data, any time ✔ Automate anything
- 61. Run workspaces on demand, on a schedule, or in response to a trigger.
- 64. FME Cloud
- 65. FME Cloud Launch Magic
- 66. Pricing change (lower for 95% of users).
- 67. New region support New! Canada East New! US-West US-East Frankfurt Sydney
- 68. New Instance Types Starter Standard Professional Premium Enterprise Previous New 4GB 8GB 16GB 32GB 30GB 3.7GB 7.5GB 15GB 30GB 30GB 30 30 55 62 1313 6.56.5 RAM Enhanced NetworkingCompute Units RAMCompute Units Yes - - - - Enhanced Networking - Yes Yes Yes Yes Burstable
- 69. FME 2017 ✔ Connect more systems ✔ Perform powerful transformations ✔ Create workflows quickly and easily ✔ Inspect any data, any time ✔ Automate anything
- 72. Trucs et astuces pour optimiser la performance de vos workbench
- 73. Performance Goals Create an FME solution faster than ever. Execute it faster than ever.
- 74. Performance Highlights for FME 2017
- 75. A Quick List ● Remodeled CSV Reading ● Improved Attribute Management ● Better List Handling ● Enhanced Polygon Dissolving ● Faster FeatureReader Workflows ● Empowered Reading of Data From Web Sources
- 76. FME en action
- 77. Goal: Add geographical information to the list of FME World Tour stops.
- 78. Steps 1. Read a Geographical Names CSV Dataset 2. Filter the Geographical Names Data for a List of Cities Only 3. Read the Current List of FME World Tour Events 4. Merge the FMEWT Event with its Country Code 5. Match the FMEWT Event to its Host City 6. Ensure the Correct Location is chosen for each Event
- 79. Here’s the workspace to solve that problem.
- 80. 2016 • Read Data: 1 minute 35 seconds • Read/Filter: 1 minute 36 seconds 2017 • Read Data: 8.3 seconds • Read/Filter: 14.1 seconds Processing Times
- 81. Summary Performance in Creation • Attributes to Read • List Element Selection • HTML Table Reader • Select Source from Web • Modeless Parameter Dialogs • Quick Add Readers/Writers • User Parameter Copy/Paste • Open Containing Folder Performance in Execution • Attributes to Read • List Element Selection • Feature Table Formats • Feature Table Transformers • Faster Polygon Dissolving • Faster FeatureReader
- 83. Pause-café De retour à 10h50…
- 85. Date & Time Attribute Manipulation
- 86. 1. Dates and times
- 87. RCMP E Division Heidi Lee | Robert Shultz Goal: Load GPS records into ArcGIS. Problems: ➔ Inconsistent date formats ➔ Time zones ➔ Daylight saving Dates and times are complicated.
- 88. Formatting? ● YYMMDD, HHMMSS, UTC ● Jun 2016 ● ‘on Saturday, Jan 9th 2016, 01:00 am’ & ‘+0530’ ● 2016-12-07 12:20:07.785403-05 ● 20160313020000.000 (March 13 - Daylight Saving) ● <d v="2016-12-13T00:00:00"/> (Excel) ● YYYY-MM-DD hh:mm:ss[.nnnnnnn] (SQL Server ‘datetime2’ value) Calculations? ● Date2-Date1 = How many days?
- 92. Over to FME
- 93. Time Zones FME 2017 UTC Offset e.g. -08:00 FME 2018 IANA Time Zone e.g. America/Vancouver
- 95. Last Year Attribute Management Quality Control
- 96. Southern Company Jeff DeWitt HOK Inc. David Baldacchino Goals: ➔ Test for patterns in attribute values ➔ Extract substrings from attribute values ➔ Validate strings 2. Finding patterns NGI Belgium Jan Beyen RCMP E Div. Heidi Lee
- 97. Southern Company Problem: Attribute value cleanup - MONTANA * or Sales/Other (1) HOK Inc. Problem: Extract Sheet numbers from file names - MyProject - Sheet - A512 - PARTITION TYPES & … - G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ... NGI Problem: Validate address strings - Rue Achille Masset 52A
- 98. Sheet number extraction MyProject - Sheet - A512 - PARTITION TYPES & … G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ... ^[Ss]*?[-]?[ ]?([A-Z0-9]{1,5})[ ]+[-]+[Ss]*$ Address validation Rue Achille Masset 52A ^((([a-zA-Z]+) )+)([0-9]+)([a-zA-Z]*)$ Wave the wand of regex
- 101. Regex vs. String Functions Example Code ABD3705337067 Regular Expression: ([A-Z]{3})([0-9]+) String Functions: Attribute String Function alpha @Left(@Value(Code),3) beta @Substring(@Value(Code),3,-1)
- 102. TRC Inc. Peter Veensta Goals: ➔ Compare current and previous Excel rows. ➔ Sum attribute values with the previous row. 3. Time-travelling attributes. FPInnovations Matt Kurowski
- 103. Past, Present & Future Attributes feature[-1].measure measure feature[+1].measure
- 104. Attribute Aggregation Challenge Rule: If T is 3 or less, aggregate this row with the row above.
- 106. Summary 1. DateTime transformers and Text Editor functions help with: ○ Date/time formatting ○ Calculations ○ Time zones 2. Regex and string functions help with patterns. 3. Work with current and previous attribute values in the AttributeManager.
- 108. Automation: Schemas & Data Enhancement for Dry Rot Insurance Sigbjørn Tillerli Herstad
- 109. Goal: To automate and enhance the quality of daily/weekly imports of customer data to a common schema.
- 110. Problem: Source Data Mayhem ● Codan Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates ● DNB Forsikring – CSV – 1 file – 5 attributes ● Eika Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates ● Enter Forsikring – CSV – 2 files – 14 attributes ● Frende Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates ● Gjensidige – CSV– 2 file – 20 attributes – missing AddressID, coordinates ● IF Skadeforsikring – XLS – 1 file – 10 attributes – missing owner ● Jernbanepersonalets Forsikring – CSV – 3 files – 5 attributes
- 111. Achieving Automation • Define schema to use for import to database (52+ attributes, one feature type) • AttributeFilter: Separate data streams for each company • FeatureReader: Reads the actual data • AttributeManager: Convert to common schema • Can be run at scheduled intervals when a file arrives
- 112. Achieving Quality Custom transformers to improve and split data, reused on multiple files: • SplittFornavnEtternavn: Separate firstname and lastname into 2 different attributes. • SplitTelefonOgMobiltlf: Decide if number is a cellular or landline and create 2 different attributes. • SplitStreetNameNumberLetter: You have one attribute in which contains streetname, housenumber, houseletter. Output is 3 different attributes.
- 113. Achieving Quality Use existing services and databases to look up and verify values: • CheckAIDToOwner: Checks if this is the official owner of that property. • NorkartGeocoder: API to check the validity of an address, handles misspellings, validates postal number, municipality number, etc. Fresh data every day!
- 114. Achieving a Common Schema Translate each customer’s schema to the common schema: AttributeManager One separate AttributeManager for each company.
- 118. Objectifs • Comparer deux versions d’un même workspace FME. • Identifier les changements dans un nouveau workspace FME.
- 119. Démo
- 120. Merci!
- 121. Capsule #2 Chargement et téléchargement de données (Data Upload/Download)
- 122. Objectif Offrir un outil de chargement/ téléchargement de données via une interface Web simple et conviviale
- 124. Chargement de données avec FME Server (Data Upload)
- 127. Téléchargement de données avec FME Server (Data Download)
- 130. Répeption des paramètres : Data Upload Paramètres entrés par l’usager • Fichier (déposé sur le fileshare) • Addresse courriel de renvoi
- 131. Répeption des paramètres : Data Download Paramètres entrés par l’usager • Fichier (déposé sur le fileshare) • Addresse courriel de renvoi
- 132. Workspaces FME: Master / Child
- 133. 1- Réception des paramètres 3- Vérification Anti-Virus 2- Envoi du courriel de réception 4- Traitement du Workspace “Child” 5- Envoi du courriel des résultats Aperçu du Workspace
- 134. Questions?
- 135. Capsule #3 Validation et correction topologique
- 136. Objectifs • Faciliter l’intégration de données provenant de diverses sources dans une base de données spatiales. • Identifier et corriger les erreurs topologiques des intrants avant l’intégration finale.
- 137. Aperçu du processus
- 139. Aperçu du Fichier Excel: Transformation 4 onglets obligatoires: • FeatureMapping • Domains • Operations • Data
- 140. Aperçu du Fichier Excel:Validation/Correction Onglets fixes pour chaque règle à valider:
- 146. Workspace de correction : Ajout de points et attributs
- 147. Merci!
- 148. Dîner De retour à 13h00…
- 149. FME Server: Charms for the Enterprise
- 150. Enterprise-Level Charms ★ Security - a new level with new charms ★ Deployment - move solutions easily ★ Automate the enterprise ★ Control over jobs
- 151. Security Charm #1 Active Directory
- 152. Any number of Active Directory domains Any number of users per domain A hybrid model - mix server and AD users Active Directory Functionality based on feedback and experience from users! Thank you!
- 153. Active Directory Support Import Active Directory Users Hybrid security model
- 154. Over to FME
- 155. Security Charm #2 Sharing and user control
- 156. ✓ Users can share with other users or groups. ✓ Sharing defined for everything except “shared resources”. ✓ Users are isolated from each other. ✓ Administration load is low. Familiar Sharing Model (Google, Dropbox, etc.) Modern Sharing Model
- 157. User-Controlled Security ✓ Users have control of all FME Server components ✓ Similar to state-of-the-art services like Google Drive.
- 158. Security Charm #3 Connections & 3rd-Party Applications
- 159. Connection Based 3rd Party Integration Why save connections? ● Encapsulate user credentials in a secure manner. ● Define for Databases and Web Applications ● Use across multiple workspaces. ● Share without revealing credentials.
- 160. Connection Based 3rd Party Integration Many web connections. ● Amazon Web Services ● OAuth 2.0 (tens of thousands of services) ● Token ● HTTP Authentication All FME-supported databases.
- 161. Database Connections
- 162. Web Connections Many web connections are shipped with FME
- 163. Web Connections More are being added to FME Hub all the time
- 164. Security Charm #4 We secure your infrastructure too.
- 165. On Premises vs FME Cloud On Premises: ● You secure your infrastructure ● You configure FME Server security FME Cloud: ● You configure FME Server security
- 166. FME Cloud Security We manage data, application and transmission security. ● HTTPS only ● Annual security audit ● Automated security patching ● Leverage AWS Compliance - Industry leaders ● Control firewall rules (protocols & ports)
- 167. Deployment Charm Easily move solutions
- 168. Challenge: How can you move an FME Server solution from one server to another?
- 169. Your FME Server solutions usually consist of many components. Workspaces Subscriptions Schedules Publications Resources Connections Custom Formats Custom Transformers Topics
- 170. FME Server Projects Portable Server Solutions Place your screenshot here
- 171. FME Server Projects Portable way to move, distribute, and share FME Server solutions. Benefit #1 - Move a project from one machine to another. Development StagingTest Production
- 172. Projects. Benefit #2 - Backup (export) a project any time for archiving. FME Server Projects
- 173. Projects. Benefit #3 - Share your projects. First - Export your project! Projects. Project Export FME Server Projects
- 174. Projects. Benefit #3 - Share your projects Next - Send to those with whom you want to share it. Projects. Project Export FME Server Projects
- 175. Projects. Benefit #3 - Share your projects Finally - Receivers simply import the project. Projects. Project Export FME Server Projects
- 176. Projects. Benefit #4 - Publish Projects to FME Hub. (Coming Soon) Share FME Server solutions with the FME community. Projects. Project Export FME Server Projects
- 177. Automation Charm
- 178. Enterprise Automation This charm is so powerful that we have another presentation just on this.
- 179. Termination Charm
- 180. Goal: Ensure FME Engines don’t get stuck on a job. Workflows are often dependent on 3rd Party applications/services. An engine may hang or take too long waiting on responses.
- 181. For any job you now have a new setting (Schedules and run Workspace) Max elapsed time job will run. If timer expires job is killed! Charm: Specify maximum time a job can run.
- 182. Charm: Specify maximum time a job can run. If job takes longer than allocated then the job is cancelled.
- 183. Summary - FME Server 2017 Charms ✓ Enterprise Level Security: Enable fine-tuned control with the new security model. ✓ Enterprise Deployment Model: Move server solutions between machines using Projects. ✓ Enterprise Automation: Automate everything.* ✓ Enterprise Level Job Control: Define max job time so the server doesn’t hang.
- 185. Capsule #4 Automatisation des données corporatives (Enterprise Automation)
- 186. Objectifs • Faciliter l’intégration de données provenant de diverses sources (internes et externes). • Ajouter une composante spatiale aux données de propriété immobilière. • Mettre à jour des services web.
- 189. Exécution quotidienne du workspace Création de rapports et envoi de notification par courriel. Place your screenshot here
- 191. Infractions – Matières résiduelles Photos prises sur le terrain à partir de l’application CityView. Chargement des photos directement sur le serveur. Problème de taille des photos.
- 193. Ultimate Real-Time Monitor Anything, Update Anything
- 194. FME Server Notification Service lets you act on events as they happen, and send information as it becomes available. Publications and Subscriptions let you publish to and monitor different systems and data.
- 195. Notifications What they are for What they are not for
- 196. Notifications What they are for A brief message, usually to trigger an action. What they are not for x Transmitting large amounts of spatial data.
- 197. Notifications What they are for A brief message, usually to trigger an action. Triggering an FME Server response to an event that happened outside of FME. What they are not for x Transmitting large amounts of spatial data. x Triggering an FME Server response to a continuous series of messages (many per second).
- 198. Notifications What they are for A brief message, usually to trigger an action. Triggering an FME Server response to an event that happened outside of FME. Sending a message about something that happened in FME. What they are not for x Transmitting large amounts of spatial data. x Triggering an FME Server response to a continuous series of messages (many per second). x Sending more than one message per second about what’s happening in FME.
- 199. FME Server Notification Service See the list of what built-in publications and subscribers can be: http://fme.ly/protocols
- 200. Workflows for “Ultimate Real-Time” Poll it Push it Update it
- 201. Poll it “Are we there yet?”
- 202. Polling Workflow 1. Change detection ○ GeoRSS reader ○ Caching ○ Timestamps 2. Run the workspace on a schedule Timestamp
- 204. GeoRSS Reader
- 205. GeoRSS Reader • Contains change detection logic • Option to only read new features
- 206. Caching • Whole datasets or timestamps • ChangeDetector • SQL - match on ID
- 208. SQL to find different features: @Value(uuid) = "uuid" AND @Value(aqi) != "aqi" SQL Example
- 209. Timestamps ● Cache the timestamp ● Last updated attribute ● Filter features newer than timestamp ● Add a timestamp attributes to your dataset
- 210. Reads last cached time and replaces with current time
- 211. Checks against timestamp and filters out old features
- 212. Pros and Cons Caching whole datasets Easy to set up and detect changes. x Need to store entire dataset - processing time? Caching timestamps No storage needed. x Harder to set up - variables, datetimes. Consider: Does my dataset have a timestamp? Can I rely on the timestamp to indicate change?
- 216. ✓ Poll Waze feed ✓ Parse XML ✓ Filter ✓ Transform ✓ Validate ✓ Email notifications
- 217. Workflows for “Ultimate Real-Time” Poll it Push it Update it
- 218. Push It
- 219. Pushing Methods • Database triggers • Webhooks In FME Server: • Direct URL • REST API
- 220. Database Triggers ● Databases can trigger events. ● Functions with HTTP requests can push data to FME Server.
- 221. Database Triggers
- 222. Webhooks ● i.e. Give systems/services a URL that will respond. ● Send the notification to FME Server. ● Queue the received notifications and process the requests on a separate thread.
- 223. Pros and Cons Database Triggers Pushes right to FME Server topic. Can push lots of changes to a holding table. Perfect for real-time database changes. x Database permissions. Webhooks No need to waste effort polling. Truly real time. x Complex to set up. x Not all systems provide or accept webhooks.
- 224. Direct URL Systems can use HTTP to push data to topics or trigger workspaces to run.
- 226. REST API In-depth interaction with FME Server ● Send/receive notifications ● Run jobs ● Manage connections ● Manage users ● Licensing ● Backups ● Scheduling ● Etc!
- 227. Workflows for “Ultimate Real-Time” Poll it Push it Update it
- 228. To Poll or Push?
- 229. Poll vs. Push Considerations ● Simplicity vs. “real-time”ness ● Can the system push to FME Server? ● Are jobs too long for polling? ● API restrictions for polling?
- 230. Tip: Use FME Server for internal systems and FME Cloud for external systems.
- 231. Update It
- 232. Updating Methods • FME Writers o Bulk drop/load/overwrite o Update option • SQLExecuter • HTTPCaller
- 233. FME Writers: UPDATE operation
- 234. INSERT, UPDATE or DELETE in the middle of a workflow.
- 235. POST data or changes to systems.
- 236. Summary • Polling workflow: o Change detection + scheduling • Pushing options: o Database triggers, webhooks o FME Server: URL or REST API • Updating: o Operations in writer parameters o SQLExecutor o HTTPCaller
- 237. Questions? Database triggers tutorial: fme.ly/dbtriggers Webhooks etc: fme.ly/web REST API documentation: fme.ly/restv3
- 239. Jeu interactif The Ghost in the Machine
- 240. Prizes! Let’s practice the spells we learned today and build a workspace.
- 241. Microsoft Cognitive Services Computer Vision API - v1.0 Website and API Doc
- 242. The Game: Can you think like a machine?
- 243. Over to FME ... Place your screenshot here
- 244. Merci!
- 245. Resources ● Slides safe.com/presentations ● Blog blog.safe.com ● Community knowledge.safe.com ● Social media @safesoftware