Efficiently Computing Significant Terms in SQL

I was introduced to ElasticSearch significant terms aggregation a while ago and was positively surprised how good and relevant this metric turns out to be. For those not familiar with it, it's quite a simple concept - for a given query (foreground set) a given property is scored against the statistical significance of the background set.

For example, if we were querying for the most significant crime types in the British Transport Police:

C = 5,064,554 -- total number of crimes
T =    66,799 -- total number of bicycle thefts
S =    47,347 -- total number of crimes in British Transport Police
I =     3,640 -- total number of bicycle thefts in British Transport Police

Ordinarily, bicycle thefts represent only 1% of crimes (66,799/5,064,554) but for the British Transport Police, who handle crime on railways and stations, 7% of crimes (3,640/47,347) is a bike theft. This is a significant seven-fold increase in frequency.

The significance for "bicycle theft" would be [(I/S) - (T/C)] * [(I/S) / (T/C)] = 0.371...

Where:

• C is the number of all documents in the collection
• S is the number of documents matching the query
• T is the number of documents with the specific term
• I is the number of documents that intersect both S and T

---

For practical reasons (the sheer amount of data I have and huge ElasticSearch memory requirements), I'm looking to implement the significant terms aggregation in SQL or directly in code.

I've been looking at some ways to potentially optimize this kind of query, specifically, decreasing the memory requirements and increasing the query speed, at the expense of some error margin - but so far I haven't cracked it. It seems to me that:

• The variables C and S are easily cacheable or queriable.
• The variable T could be derived from a Count-Min Sketch instead of querying the database.
• The variable I however, seems impossible to derive with the Count-Min Sketch from T.

I was also looking at the MinHash, but from the description it seems that it couldn't be applied here.

Does anyone know about some clever algorithm or data structure that helps tackling this problem?