biongroup.blogg.se - Athena json query

#Athena json query how to#

Not withstanding the frustration of the visualization tool not updating the charts (which our Data Analysts wanted desperately), the cost was also prohibitive as Amazon Athena’s pricing is based on the bytes scanned. The query time most often exceeded the threshold set by Superset resulting in the charts on Superset not getting updated. We use Superset (visualization tool) for analyzing a dataset by querying on AWS Athena (the data being stored in AWS S3).

Optimized use of JOINS, ORDER BY and GROUP BY clauses.

Use of subqueries when required, as writing a query without nesting causes a massive data explosion, nested query is a tool for performing operations in multiple steps.

Use of (database) partitions to filter the targeted data, this helps reduce the amount of data scanned per query.

Avoiding multiple joins and if this doesn’t suffice your result, change the database table schema in a way that requires less joins.

Selecting only required fields instead of * in the SELECT statement.

Choosing the right Aggregate and Scalar functions as per the query requirement.

#Athena json query how to#

So, how to optimize the queries is the question., here are some ways to optimize a query :

We can also see a detailed information on query execution plan for the query using EXPLAIN & EXPLAIN ANALYZE statement. Having a basic idea on order of execution for query statements can be helpful.