跳到主要內容

Google Trends Hacking

給我資料

最近試著在蒐集一些看起來有點用途的資料,Google Trends 看起來是一個 “滿有價值” 的資料,是 Google 蒐集的全球趨勢資料,包含最近流行的關鍵字,關鍵字與關鍵字的比較,etc.,但 Google 並沒有提供相關的 API 介面,所以要取得資料的做法我第一個想到的是 Crawler,於是順手找了一下大神的開源專案,左看右看上看下看,都沒認真看,都覺得不順手XD。

Hack Fun

剛好讀到一篇很有趣的文章-hacking the google trends api,文章內容提到了幾個有趣的 end-point,像這樣

http://www.google.com/trends/fetchComponent?hl=en-US&q=html5,jquery&cid=TIMESERIES_GRAPH_0&export=5&w=500&h=300

URI http://www.google.com/trends/fetchComponent
hl en-US
q keywords
cid TIMESERIES_GRAPH_0/RISING_QUERIES_1_0/business_and_politics
export 5
w width
h height

還有這樣 curl 大絕招

curl --data "ajax=1&htd=20131111&pn=p1&htv=l" http://www.google.com/trends/hottrends/hotItems

還有直接把這個貼在 Browser

http://hawttrends.appspot.com/api/terms/

這邊還有大神整理的參數對照表,實在棒棒der。
http://myweb.fcu.edu.tw/~mhsung/Research/InformationSystem/JSON/JSON_24.htm

Browser Dev-Tools

看完大神的示範後,發現其實有些開放的資料並不一定要特別透過 Crawler 去取得,太沒效率了。而且大部份的瀏覽器都有提供開發工具,Hacking 那些 end-point 其實滿簡單的,比如說 Chrome 的開發者工具,我以這個潮潮的頁面來做示範好了

https://www.google.com/trends/hottrends/visualize?pn=p1

image-4

就可以愉快得到一串 curl

curl 'https://www.google.com/trends/hottrends/visualize/internal/data' -H 'Referer: https://www.google.com/trends/hottrends/visualize?pn=p1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36' --compressed

就可以愉快地寫 api 了

Sample

Reference

這個網誌中的熱門文章

Spring boot Https 設定

1. 產生 SSL certificate keytool -genkey -alias tomcat -storetype PKCS12 -keyalg RSA -keysize 2048 -keystore keystore.p12 -validity 3650 Enter keystore password: Re-enter new password: What is your first and last name? [Unknown]: jerry What is the name of your organizational unit? [Unknown]: td What is the name of your organization? [Unknown]: com What is the name of your City or Locality? [Unknown]: taipei What is the name of your State or Province? [Unknown]: taiwan What is the two-letter country code for this unit? [Unknown]: tw Is CN=jarvis, OU=td, O=urad, L=taipei, ST=taiwan, C=tw correct? [no]: yes 這個 certificate 是 self-signed certificate 沒有經過第三方認證, 所以沒有公信力, 正式上線會在瀏覽器看到 連線不被信任 要有公信力的 certificate 最簡單的是 Lets Encrypt , 其他就是花一些錢找簽發 certificate 的組織 2. Enable HTTPS in Spring Boot Spring Boot 內建的 tomcat 預設 http 是 8080, Spring Boot 可以設定 http 跟 https, 但沒辦法同時存在這兩個設定, 如果要同時存在兩種 connection, 建議依照 文件 建議設定 https, 再透過 programmatically 去設定 http 會比較容易。 相關範例可以參考 : https://...

Google Compute Engine‎ - AccessDeniedExceptions 403

原因 打算從 instance 打包 logs 到 google cloud storage 發生了 AccessDeniedException: 403 Insufficient OAuth2 scope to perform this operation. , 看起來是 instance 沒有 storage 權限 解決 Reference: https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes 重新設定 service account 權限 instance 上內建有 gcloud , 就直接用現有的工具查詢一下 instance 的 account. $ gsutil info 或者在本機直接 gcloud compute instances describe INSTANCE_NAMES Account: [alpha-number-compute@developer.gserviceaccount.com] Project: [our-project-name] 會看到 instance 的一些狀態, 接下來就簡單多了, 按照下列的說明, 要先 stop instance, 更改 storage scope 再重新 start 。 To change an instance's service account and access scopes, the instance must be temporarily stopped. To stop your instance, read the documentation for Stopping an instance. After changing the service account or access scopes, remember to restart the instance. # Stop Instance gcloud compute instances stop INSTANCE_NAMES # 設定 storage scope 為 full (Read, Write) gcloud co...

Java Process And Thread 筆記

英文很差,所以附上原文。 關係大概是這樣吧 Processes And Threads Processes (進程) 每個 Process 都可視為一個獨立運作的環境,有自己完整、私有的 Run-Time 資源,特別是每個 process 都有自己的記憶體空間。 Processes 聽起來很像 programs 或 application 的另一種說法,然而事實上一個 application 有可能是許多 processes 合作組成的。為了讓 processes 彼此互相合作,一些作業系統支援了 Inter Process Communication (IPC) resources,例如 pipes 與 sockets。IPC 不只讓不同的 processes 互相通訊,同時也支援不同系統的 pocesses 互相通訊。 大部分 JVM 實現的都是 single process。Java Application 可以創建額外的 process 透過 ProcessBuilder Object。 A process has a self-contained execution environment. A process generally has a complete, private set of basic run-time resources; in particular, each process has its own memory space. Processes are often seen as synonymous with programs or applications. However, what the user sees as a single application may in fact be a set of cooperating processes. To facilitate communication between processes, most operating systems support Inter Process Communication (IPC) resources, such as pipes and sockets. IPC is used not just for communication b...