Building Search for Bitbucket Cloud
•
0 likes•10,582 views
Learn how we built search for Bitbucket Cloud using a Microservices approach, leveraging the foundation we shipped in Bitbucket Server. Hear about the challenges faced when building search for millions of users, building on top of the Bitbucket Connect platform, the approach the team took, and the lessons we learned. Stefan Saasen, Bitbucket Architect, Atlassian
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (19)
Similar to Building Search for Bitbucket Cloud
Similar to Building Search for Bitbucket Cloud (20)
More from Atlassian
More from Atlassian (20)
Recently uploaded
Recently uploaded (20)
Building Search for Bitbucket Cloud
- 1. Building Search in Bitbucket Cloud STEFAN SAASEN | ARCHITECT | ATLASSIAN | @STEFANSAASEN 1
- 2. (FunFactory OR foo.bar) NOT "no rest" ext:java 2
- 3. Simple! 3
- 4. Simple? 4
- 6. 600 TB repository data (approximately) 6
- 7. 13,000 million files (a rough estimate) 7
- 8. 87,000 index events per hour at peak 8
- 9. Search code != natural language Scale How to scale code search?Our challenges 9
- 10. Search code != natural language Scale How to scale code search?Our challenges 10
- 11. How does a search engine work? SEARCH 101 11
- 12. SEARCH 101 Content acquisition byte stream to character stream Tokenization Normalization Document Mapping & Analysis Index Query parsing & search execution Query interface Result presentation 12
- 13. SEARCH 101 Content acquisition byte stream to character stream Tokenization Normalization Document Mapping & Analysis Index Query parsing & search execution Query interface Result presentation 13
- 14. SEARCH 101 Content acquisition byte stream to character stream Tokenization Normalization Document Mapping & Analysis Index Query parsing & search execution Query interface Result presentation 14
- 15. Analysis val x = foo.Bar(user, repo); // Refactor this! 15
- 16. Analysis val foo.bar foo bar user repo refactor this 16
- 17. SEARCH 101 Content acquisition byte stream to character stream Tokenization Normalization Document Mapping & Analysis Index Query parsing & search execution Query interface Result presentation Inverted Index Central idea in information retrieval. 17
- 18. INVERTED INDEX bar foo foo.bar val 312 544 545 12 56 221 234 312 544 234 300 545 611 699 300 611 3 6 5 2 Document Frequency PostingsDictionary Sorted Sorted 18
- 20. Query foo.bar 20
- 21. INVERTED INDEX bar foo foo.bar val 312 544 545 12 56 221 234 312 544 234 300 545 611 699 300 611 3 6 5 2 Document Frequency PostingsDictionary Sorted Sorted 21
- 24. bar foo foo.bar val 312 544 545 12 56 221 234 312 544 234 300 545 611 699 300 611 3 6 5 2 Document Frequency PostingsDictionary Sorted Sorted INVERTED INDEX 24
- 25. Ranked retrieval Rank order the documents matching a query. 25
- 28. Store declarations Each file document contains a declarations fields Language aware syntax extraction Boost Boost documents where the declaration list contains the term Extract declarations Class names, methods, symbols etc. 28
- 29. Store declarations Each file document contains a declarations fields Language aware syntax extraction Boost Boost documents where the declaration list contains the term Extract declarations Class names, methods, symbols etc. 29
- 30. Store declarations Each file document contains a declarations fields Language aware syntax extraction Boost Boost documents where the declaration list contains the term Extract declarations Class names, methods, symbols etc. 30
- 32. Let's do that for billions of files! 32
- 33. Goals and Qualities Requirements Load parameters: rate of change Optimise: indexing throughput Search at scale. 33
- 34. Goals and Qualities Requirements Load parameters: query per second Optimise: query response time Search at scale. 34
- 35. Goals and Qualities Scalability, Reliability, Functionality and Security Requirements Load parameters: query per second Optimise: query response time Search at scale. 35
- 36. SYSTEM ARCHITECTURE Webhook Ingester Secret Store Indexer Syntax Extraction Query Routing Proxy 36
- 37. SYSTEM ARCHITECTURE Search Webhook Ingester Secret Store Indexer Syntax Extraction Query Routing Proxy Index Pipeline 37
- 38. Index Pipeline Ingester Secrets Indexer Syntax Event ingestion Via Bitbucket Connect Webhooks. Webhook Ingester 38
- 39. Index Pipeline Ingester Secrets Indexer Syntax Secret Store Common Connect concern. Secret Store 39
- 41. Index Pipeline Ingester Secrets Indexer Syntax Syntax extraction File content Declarations REST 41
- 42. Avoid single point of failures RELIABILITY 42
- 43. Routing & Sharding Routing Proxy Cluster 1 - n 43
- 44. Query Service User facing REST API Query Parsing Custom ANTLR based query parser. Elasticsearch query generation. Secret Store Query Routing Proxy Bitbucket API 44
- 45. Query Service User facing REST API Query Parsing Custom ANTLR based query parser. Elasticsearch query generation. Secret Store Query Routing Proxy Bitbucket API 45
- 46. 46
- 47. “A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable. LESLIE LAMPORT 47
- 48. Requirements E.g. load parameters to guide approaches Principles Identify general truths Patterns 12factor, eventual consistency, feature flags Innovation Dollars Be careful where you spend development effort 48
- 49. Requirements E.g. load parameters to guide approaches Principles Identify general truths Patterns 12factor, eventual consistency, feature flags Innovation Dollars Be careful where you spend development effort 49
- 50. Requirements E.g. load parameters to guide approaches Principles Identify general truths Patterns 12factor, eventual consistency, feature flags Innovation Dollars Be careful where you spend development effort 50
- 51. Requirements E.g. load parameters to guide approaches Principles Identify general truths Patterns 12factor, eventual consistency, feature flags Innovation Dollars Be careful where you spend development effort 51
- 52. Tooling is key Log aggregation and analysis, monitoring, distributed tracing, alerting and chaos. 52
- 53. BITBUCKET SERVER Shipped in 2016 Bitbucket Server 4.6 Server & Cloud? Shared Components 53
- 54. BITBUCKET SERVER Shipped in 2016 Bitbucket Server 4.6 Server & Cloud? Shared Components 54
- 55. BITBUCKET SERVER Shipped in 2016 Bitbucket Server 4.6 Server & Cloud? Shared Components 55
- 56. 56
- 57. Server & Cloud? Components SPI Connect Constraints Shared components Libraries and components 57
- 58. Server & Cloud? Components SPI Connect Constraints Service Provider Interface (SPI) Content acquisition, permissions, scope resolution. 58
- 59. Server & Cloud? Components SPI Connect Constraints Atlassian Connect Loose coupling allows for freedom of implementation. 59
- 60. Server & Cloud? Components SPI Connect Constraints Design constraints Biased towards Cloud constraints. 60
- 61. Overhead Technical solution versus team and roadmap alignment. Possible Cloud and server codebases diverged. Cross porting changes, common library. Build once, Ship twice? Suitable? Entirely different operational concerns need to be accounted for. 61
- 62. Overhead Technical solution versus team and roadmap alignment. Possible Cloud and server codebases diverged. Cross porting changes, common library. Build once, Ship twice? Suitable? Entirely different operational concerns need to be accounted for. 62
- 63. Overhead Technical solution versus team and roadmap alignment. Suitable? Entirely different operational concerns need to be accounted for. Possible Cloud and server codebases diverged. Cross porting changes, common library. Build once, Ship twice? 63
- 64. 64
- 65. STEFAN SAASEN | ARCHITECT | ATLASSIAN | @STEFANSAASEN Thank you! 65