2017

計算 Mysql database 大小

參考: https://www.mkyong.com/mysql/how-to-calculate-the-mysql-database-size/


Sql

SELECT TABLE_SCHEMA, SUM(DATA_LENGTH + INDEX_LENGTH)/1024/1024 "DATA_SIZE(MB)"
FROM information_schema.TABLES 
GROUP BY TABLE_SCHEMA;

GROUP BY 可以替換成 WHERE TABLE_SCHEMA= your-data-base-name

SELECT TABLE_SCHEMA, SUM(DATA_LENGTH + INDEX_LENGTH)/1024/1024 "DATA_SIZE(MB)"
FROM information_schema.TABLES 
WHERE TABLE_SCHEMA=erp;

LENGTH(str) 用途

參考:https://dev.mysql.com/doc/refman/5.5/en/string-functions.html#function_length

Returns the length of the string str, measured in bytes. A multibyte character counts as multiple bytes. This means that for a string containing five 2-byte characters, LENGTH() returns 10, whereas CHAR_LENGTH() returns 5.

簡單翻譯 LENGTH() 會計算 string 的 byte.

SELECT LENGTH('hello_world')
# 回傳 11

計算 all table 某個欄位大小

SELECT SUM(LENGTH(`column_name`)) 
FROM table_name
WHERE id = 1234567;

計算 one raw data 大小

SELECT (LENGTH(`id`) + LENGTH(`column_1`) + LENGTH(`column_2`) + LENGTH(`column_3`) + LENGTH(`column_4`)) RAW_SIZE
FROM table_name
WHERE id = 1234567;

Parse URI query String to Map

  • 做 urlDecode 處理
  • 沒有任何 query String 回傳 Empty Map
  • 確保只處理 key-value 結構的 query String


package com.example.util;

import lombok.extern.slf4j.Slf4j;
import org.apache.http.client.utils.URIBuilder;

import java.io.UnsupportedEncodingException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URLDecoder;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Objects;

/**
 * Created by jerry on 2017/12/28.
 *
 * @author jerry
 */
@Slf4j
public class UriUtil {

    private UriUtil() {
    }

    public static Map splitQuery(final String uri) {

        Map queryPairs = new LinkedHashMap<>();

        try {
            final URI uri = new URIBuilder(uri).build();
            final String rawQuery = uri.getRawQuery();
            log.info("CurrentUrl Query: {}", rawQuery);

            // 過濾沒有 query string
            // 還有過濾無法成對 keyValue 的 query, e.g. http://host/path?123
            if (Objects.isNull(rawQuery) || !rawQuery.contains("=")) {
                return queryPairs;
            }
     
            final String[] pairs = rawQuery.split("&");

            final String utf8 = "UTF-8";

            for (String pair : pairs) {

                // 統一 decode
                final String deCodePair = URLDecoder.decode(pair, utf8);

                // check deCodePair 空字串,
                if (!deCodePair.isEmpty()) {
                    final String[] keyValue = pair.split("=");

                    // check is key value
                    if (2 == keyValue.length) {
                        final String key = keyValue[0];
                        final String value = keyValue[1];

                        queryPairs.put(key, value);
                    }
                }
            }

            log.info("Parser url: {} \n parameters: {}", url.toString(), queryPairs);
            return queryPairs;

        } catch (URISyntaxException uriBuilderUrlException) {
            uriBuilderUrlException.printStackTrace();
            log.warn("UriBuilder url ({}) has exception: {}", url, uriBuilderUrlException.getMessage());

        } catch (UnsupportedEncodingException urlDecodeException) {
            urlDecodeException.printStackTrace();
            log.warn("Decode url ({}) query has exception: {}", url, urlDecodeException.getMessage());
        }

        return queryPairs;
    }
}

快速參考

https://cloud.google.com/sdk/gcloud/reference/config/

gcloud config 建立

Step1. 建立設定檔, gcloud config configurations create

$ gcloud config configurations create your_config_name

Step2. gcloud init 會有引導, 協助建立 gcloud config 細節(mapping project name, region, etc.)

$ gcloud init

gcloud config 切換

$ gcloud config configurations activate your_config_name

Sample Code

String str = "a,b,c,,";
String[] ary = str.split(",");

// result 3
System.out.println(ary.length);

在 code review 的時候發現一個有趣的狀況,
split(",") 處理完, 預期結果應該是 5, 然而實際上卻是 3,
查了一下 API 發現 public String[] split(String regex) 要求的參數其實是 regrex express,
這有可能導致 IndexOutOfBoundException, 建議改為

public String[] split(String regex, int limit)

limit 用來限制 array 的長度,
如果 limit > 0, 最終處理的 array 長度不會大於 limit, regex express 匹配的次數最多為 n - 1 次,
如果 limit < 0, regex express 會盡可能的處理匹配, 包含對 空字串 匹配的問題,
如果 limit = 0, regex express 會儘可能地處理匹配, 但會放棄處理 空字串 匹配的問題。

參考:API Doc

Sample Code

String str = "a,b,c,,";
String[] ary = str.split(",", -1);

// result 5
System.out.println(ary.length);

測試驅動開發(TDD)引發的爭論

這是一篇來自於 import new 的探討: 測試驅動開發(TDD)引發的爭論,
還有 Kent Beck 在 StackOverFlow 的解釋:

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don't typically make a kind of mistake (like setting the wrong variables in a constructor), I don't test for it. I do tend to make sense of test errors, so I'm extra careful when I have logic with complicated conditionals. When coding on a team, I modify my strategy to carefully test code that we, collectively, tend to get wrong.

Different people will have different testing strategies based on this philosophy, but that seems reasonable to me given the immature state of understanding of how tests can best fit into the inner loop of coding. Ten or twenty years from now we'll likely have a more universal theory of which tests to write, which tests not to write, and how to tell the difference. In the meantime, experimentation seems in order.

我自己覺得, 測試價值不應該著重在追求覆蓋率, 而是減少高複雜度的代碼出錯的機會, 應該是上面這段話的核心價值, 另外在團隊合作開發的過程中 TDD 也是讓合作人員快速了解某些模塊的功能, 在 code Reveiew 之後多建立一道防線來保證代碼的品質。


Atom

突然想學 ES6, 所以開始研究了 Atom, 先參考別人的設定。
Reference: https://github.com/farrrr/atom

Plugins

Theme 簡單地用了 atom 預設的 atom-dark-ui,
其他的 plugins 雖然介紹很多, 但還是覺得先不要用太多妖魔鬼怪, 避免分散力氣去解決地雷。


ReactJS Framework

就單純的隨便找一個 Framework 來當作起手, 訴求是學習資源多的 Framework, Vue.js 好像也不錯。

ReactJS HelloWorld

# 使用 nvm 切換到 stable 的 node.js 版本
$ nvm use stable
# 在 global 安裝 create-react-app module
$ npm install -g create-react-app
# 建立 react-hello-world 專案
$ create-react-app react-hello-world
# 在 local 啟動專案
$ cd react-hello-world
$ yarn start

Preview


首先

這篇主要是翻譯 這篇 的教學, 再用自己的方式解釋。

Email Service

文章開頭以 Email Service 來解釋, UML 應該是長這樣, Application 強依賴 EmailService, Application 在一開始就初始化了 EmailService, 而 Main 也強依賴 Application, 這樣寫並沒有太大的問題, 但不容易對 Application class 做測試, 因為不容易對 EmailService 做 mock.



修正 Application 方便測試

為了讓 Application 能夠被測試, 所以改成 inject EmailService 的建構方式




public class Application {

    private EmailService emailService;

    public Application(EmailService emailService) {
        this.emailService = emailService;
    }

    public void notification(String message, String receiver) {
        this.emailService.sendEmail(message, receiver);
    }
}

簡單的使用 Mockito 來 mock EmailService 測試

<dependency>
    <groupId>org.mockito</groupId>
    <artifactId>mockito-core</artifactId>
    <version>2.12.0</version>
</dependency>


package dependency.injection;

import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;

import org.junit.Before;
import org.junit.Test;
import org.mockito.Mock;
import org.mockito.MockitoAnnotations;

/**
 * Created by jerry on 2017/11/21.
 */
public class ApplicationTest {

    @Mock
    private EmailService mockEmailService;

    @Before
    public void setUp() {
        MockitoAnnotations.initMocks(this);
    }

    @Test
    public void assertApplication() {
        Application app = new Application(mockEmailService);

        // trigger
        app.notification("test-message", "someone");

        // mock
        doNothing()
                .when(mockEmailService)
                .sendEmail(anyString(), anyString());

        // verify emailService will be call once
        verify(mockEmailService, times(1));
    }

}

MessageService Interface

Interface 的好處, 是讓我們方便活用物件導向的多型(Polymorphism)概念, 透過 MessageService Interface, 讓 EmailServiceSmsService 擁有一致性的操作 void sendMessage(), 好處是 Application 不再依賴一個明確的 class instance, 而是依賴一個抽象的 interface, 對 Application 來說, 只要能做 sendMessage 的物件都可以接受, Application 不再 “強烈” 依賴 EmailService 或是 SmsService.

日後擴增 FacebookMessage 只要 implement MessageService, 也一樣可以被 Application 使用.




public interface MessageService {

    void sendMessage(String message, String receiver);
}


public class EmailServiceImpl implements MessageService {

    @Override
    public void sendMessage(String message, String receiver) {
        System.out.println("Email sent to " + receiver + " with Message=" + message);
    }
}


public class SmsServiceImpl implements MessageService {

    @Override
    public void sendMessage(String message, String receiver) {
        System.out.println("SMS sent to " + receiver + " with Message=" + message);
    }
}


public class Application {

    private MessageService messageService;

    public Application(MessageService messageService) {
        this.messageService = messageService;
    }

    public void notification(String message, String receiver) {
        this.messageService.sendMessage(message, receiver);

    }
}


public class Main {

    public static void main(String[] args) {
        Application app = new Application(new EmailServiceImpl());
        app.notification("nice to meet you", "someone");
    }
}

Consumer

文章末端, 用了 Consumer interface 抽象了原本的 Application, 而 consumer 做的也只是 sendMessage 這個動作, 多墊一個 interface 好處並不多, 反而增加多餘的 code 造成維護上的不便.


優點

  1. 關注點分離(Separation of Concerns), 對 Application class 來說只需要關注 sendMessage 的動作.
  2. 方便測試, 對 Application 來說, 只要知道 sendMessage 有被執行, 執行的內容不是重點.
  3. 解耦合, Application 不強烈依賴任何 instance.
  4. 操作反轉(IoC), Application 不在需要在操作的時候決定要用 EmailService 還是 SmsService, 而交由外部 class (Main)決定.

缺點

  1. 維護跟擴增並不一定容易, 假設 FacebookMessage 的 sendMessage 是需要互動的情境, 有回傳值 String sendMessage(), 那應該是寫 2 個 interface ? 還是在相同的 interface 寫 2 個 methods ?
  2. 單純的 Code Review 的階段, 無法了解是被 injection 是哪個 instance ? EmailServiceImpl or SmsServiceImpl ?


Liquibase Migration

Laravel 的 Migration 是一個很棒的工具,
可以簡單的下個指令, 輕鬆的建立 tables
php artisan migrate

在 Spring-boot 也提供一套類似的工具, 使用起來大同小異, 只是設定上有點小麻煩。

dependencies


<dependency>
    <groupId>org.liquibase</groupId>
    <artifactId>liquibase-core</artifactId>
</dependency>

build


<build>
    <resources>
        <resource>
            <directory>src/main/resources</directory>
        </resource>
    </resources>
    
    <testResources>
        <testResource>
            <directory>src/test/resources</directory>
        </testResource>
    </testResources>

    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.6.2</version>
            <configuration>
                <source>${java.version}</source>
                <target>${java.version}</target>
            </configuration>
        </plugin>

        <!--用來快速產生 migration-->
        <plugin>
            <groupId>org.liquibase</groupId>
            <artifactId>liquibase-maven-plugin</artifactId>
            <version>3.5.3</version>
            <configuration>
                <url>jdbc:mysql://localhost:3306/liquibase_demo</url>
                <changeLogFile>src/main/resources/config/liquibase/master.xml</changeLogFile>
                <driver>com.mysql.jdbc.Driver</driver>
                <username>username</username>
                <password></password>
            </configuration>
        </plugin>
    </plugins>
</build>

application.yml



# database
database:
    host: localhost
    name: liquibase_demo
    username: root
    password:
spring:
    profiles:
        active: dev
    application:
        name: urad-facebook-data
    datasource:
        username: ${database.username}
        password: ${database.password}
        url: jdbc:mysql://${database.host}:3306/${database.name}?characterEncoding=utf-8&useUnicode=true&useSSL=false&rewriteBatchedStatements=TRUE

# liquibase configuration
liquibase:
    enabled: true
    change-log: classpath:config/liquibase/master.xml
    url: ${spring.datasource.url}
    user: ${spring.datasource.username}
    password: ${spring.datasource.password}

Changle Log

Change Log 的路徑, Srping-boot 預設為 db/changelog/db.changelog-master.yaml,
可以參考 文件, 我為了方便管理設定為 src/main/resources/config/liquibase/changelog/master.xml

Create User Table

<?xml version="1.1" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" 
                      xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext" 
                      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
                      xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext 
                                         http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd 
                                         http://www.liquibase.org/xml/ns/dbchangelog 
                                         http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
    <changeSet author="jerry (generated)" id="1511234719817-1">
        <createTable tableName="user">
            <column autoIncrement="true" name="id" type="INT UNSIGNED">
                <constraints primaryKey="true"/>
            </column>
            <column name="name" type="CHAR(32)"/>
            <column name="age" type="INT"/>
        </createTable>
    </changeSet>
</databaseChangeLog>

將 change log 添加到 master.xml

<?xml version="1.0" encoding="utf-8"?>
<databaseChangeLog
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog 
                       http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd">

    <include file="classpath:config/liquibase/changelog/00000000000000_initial_user_schema.xml" relativeToChangelogFile="false"/>
</databaseChangeLog>

Liquibase plugin 基本指令

liquibase:update: 將新的 change log 更新到目前的資料庫.

mvn liquibase:update

liquibase:generateChangeLog: 依據現有的資料庫的狀態, 產生 liquibase 的 Change log.

mvn liquibase:generateChangeLog

liquibase:dbDoc: 依據現有的資料庫的狀態, 產生 db doc 文件, 類似 Java doc, 參考 範例,
預設產出的文件路徑 target/liquibase/dbDoc/index.html

mvn liquibase:dbDoc

liquibase 文件



Quartz

Quartz 是一個功能滿完整的 Java Schedule 排程工具, 核心就圍繞在 Scheduler 與 Job 的操作上。

  • Scheduler - the main API for interacting with the scheduler.
  • Job - an interface to be implemented by components that you wish to have executed by the scheduler.
  • JobDetail - used to define instances of Jobs.
  • Trigger - a component that defines the schedule upon which a given Job will be executed.
  • JobBuilder - used to define/build JobDetail instances, which define instances of Jobs.
  • TriggerBuilder - used to define/build Trigger instances.


Job

Quartz 的 Job 被定義為需要 implement org.quartz.Job,
需要實作 void execute(JobExecutionContext jobExecutionContext),
jobExecutionContext, 可以取得 Job 的 Scheduler, Trigger, JobDetail 的相關設定, 相關的說明可以參考 tutorial-lesson-02.


import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.quartz.Job;
import org.quartz.JobDataMap;
import org.quartz.JobDetail;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;

/**
 * Created by jerry on 2017/11/17.
 */
public class HelloJob implements Job {

    /**
     * Quartz Job 要求要一個 empty constructor,
     * 讓 Scheduler 來 instantiate
     */
    public HelloJob() {
    }

    /**
     * Quartz Job 要執行的動作
     *
     * @param jobExecutionContext
     * @throws JobExecutionException
     */
    @Override
    public void execute(JobExecutionContext jobExecutionContext) throws JobExecutionException {
        ObjectMapper objectMapper = new ObjectMapper();
        JobDetail jobInstant = jobExecutionContext.getJobDetail();
        System.out.println("===========================================");
        System.out.println("Description: " + jobInstant.getDescription());
        System.out.println("Job Key: " + jobInstant.getKey());
        System.out.println("is Concurrent Execution Disallowed: " + jobInstant.isConcurrentExectionDisallowed());
        System.out.println("is Durable: " + jobInstant.isDurable());
        System.out.println("requests Recovery: " + jobInstant.requestsRecovery());

        System.out.println("===========================================");
        JobDataMap jobData = jobInstant.getJobDataMap();

        try {
            String jobDataJson = objectMapper.writeValueAsString(jobData);
            System.out.println("Job Data Map Data: " + jobDataJson);

        } catch (JsonProcessingException e) {
            e.printStackTrace();
        }

    }
}


Scheduler

定義完 Job 之後, 要觸發還需要 Scheduler, Scheduler 可以用來分配 Job 與 Trigger,
藉由 SchedulerFactory 來取得 instance, SchedulerFactory 會載入 quartz.properties 定義的相關設定(參考


import static org.quartz.DateBuilder.evenMinuteDate;
import static org.quartz.JobBuilder.newJob;
import static org.quartz.TriggerBuilder.newTrigger;

import com.google.common.collect.ImmutableMap;
import org.quartz.JobDataMap;
import org.quartz.JobDetail;
import org.quartz.Scheduler;
import org.quartz.SchedulerException;
import org.quartz.SchedulerFactory;
import org.quartz.Trigger;
import org.quartz.impl.StdSchedulerFactory;

import java.util.Date;

/**
 * Created by jerry on 2017/11/17.
 */
public class HelloWorldSchedule {
    
    public void run() throws SchedulerException, InterruptedException {
        // init job
        JobDetail job = newJob(HelloJob.class)
            .withDescription("quartz job - hello world")
            .withIdentity("job-name", "job-group")
            .usingJobData("string-key", "value1")
            .usingJobData("long-key", 1L)
            .usingJobData(new JobDataMap(
                ImmutableMap.builder().put("map-key", "map-value").build()))
            .build();

        // set trigger time
        Date runTime = evenMinuteDate(new Date());

        // init trigger, start at next even minute time
        Trigger trigger = newTrigger()
            .withIdentity("trigger-name", "trigger-group")
            .startAt(runTime)
            .build();

        // init schedule
        SchedulerFactory schedulerFactory = new StdSchedulerFactory();
        Scheduler scheduler = schedulerFactory.getScheduler();
        scheduler.scheduleJob(job, trigger);

        // start up schedule
        scheduler.start();

        // sleep 10 seconds, make sure the scheduler be triggered
        Thread.sleep(60L * 1000);

        scheduler.shutdown(true);
    }
}

Run Scheduler


@Test
public void testScheduler() throws SchedulerException, InterruptedException {
    HelloWorldSchedule helloWorldSchedule = new HelloWorldSchedule();
    helloWorldSchedule.run();
}

覺得 String.format("%1$02x", b) 很神奇, 隨手記錄一下

String to Hex


public static String toHex(byte[] bytes) {
    StringBuilder sb = new StringBuilder();
    for (byte b : bytes) {
        sb.append(String.format("%1$02x", b));
    }
    return sb.toString();
}

Hex to String


import java.io.UnsupportedEncodingException;
import javax.xml.bind.DatatypeConverter;

public static String hexToString(String hexStr) throws UnsupportedEncodingException {
    byte[] bytes = DatatypeConverter.parseHexBinary(hexStr);
    return new String(bytes, "UTF-8");
}

網路管理基礎

Route53 與網路管理基本概念
EC2 Https 設定

AWS 綁定 Domain

  1. Route53 建立 Hosted Zones, 自己申請的 Domain (my-domain.me)
  2. Route53 Hosted Zones 建立 Record (my-domain.me), 綁定 EC2

Let’sEncrypt Certificate 申請

1. 安裝 certbot 工具

$ git clone https://github.com/certbot/certbot.git
$ cd certbot
$ ./certbot-auto

2. 透過 certbot 的 plugin 申請 certificate


$ ./certbot-auto certonly —a standalone -d my-domain.me —email someone@gmail.com 

在流程中, 需要回答一些問題, LetsEncrypt 會發送 request 去驗證 my-domain.me 這個 domain


3. 使用 openssl 將 letsEncrypt 的 key 產生 ssl

這是 os 環境 letsencrypt 的預設目錄路徑

$ cd /etc/letsencrypt/live/my-domain.me

把 letEncrypt 給你的 private key 產生 ssl 憑證

$ openssl pkcs12 -export -in fullchain.pem \ 
                 -inkey privkey.pem \ 
                 -out keystore.p12 
                 -name tomcat \
                 -CAfile chain.pem \
                 -caname root

把生成的 ssl 憑證, 放到 spring-boot 專案底下的 resouces


$ cp keystone.p12 ~/my-web-project/main/resources/

4. 設定 application-prod.yml


server:
    port: 8443
    ssl.key-store: keystore.p12
    ssl.key-store-password: pwd
    ssl.keyStoreType: PKCS12
    ssl.keyAlias: tomcat

5. Http Redirect to Https

Http 已經退流行了, Google 也不歡迎 Http, 就順手把 Http 導向到 Https 吧,
@profile("prod"), 的目的是讓本地開發(dev)流程簡單一點, 不走 Https,
只有機器上線(prod)才讓這個 Configuration Bean 被啟動.


@Profile("prod")
@Configuration
public class WebConfiguration {

    @Value("${server.port}")
    private Integer serverPort;

    @Bean
    public EmbeddedServletContainerFactory servletContainer() {
        TomcatEmbeddedServletContainerFactory tomcat = new TomcatEmbeddedServletContainerFactory() {
            @Override
            protected void postProcessContext(Context context) {
                SecurityConstraint securityConstraint = new SecurityConstraint();
                securityConstraint.setUserConstraint("CONFIDENTIAL");
                SecurityCollection collection = new SecurityCollection();
                collection.addPattern("/*");
                securityConstraint.addCollection(collection);
                context.addConstraint(securityConstraint);
            }
        };

        tomcat.addAdditionalTomcatConnectors(initiateHttpConnector());
        return tomcat;
    }

    private Connector initiateHttpConnector() {
        Connector connector = new Connector("org.apache.coyote.http11.Http11NioProtocol");
        connector.setScheme("http");
        connector.setPort(8080);
        connector.setSecure(false);
        connector.setRedirectPort(serverPort);

        return connector;
    }
}

6. Package and Deploy


$ mvn clean package


$ curl https://my-domain.me:8443/

Let’sEncrypt Certificate 限制

  • Names/Certificate:單一 certificate 限制 100 個 hostname。
  • Certificates/Domain:每個 domain 每個禮拜最多 20 個 certificate,但 renew 不計算在 quota 內 (需要憑證內的 hostname 與之前完全一樣)。
  • Certificates/FQDNset:相同 hostname 的憑證每個禮拜最多發出五個。

Renew Certificate

因為有安裝 certbot Renew 這個動作, 它也包裝好了, 一鍵使用如下


$ cd certbot
$ ./certbot-auto

這邊是詢問, 要處理哪個 domain


Requesting to rerun /bin/certbot-auto with root privileges...
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
-------------------------------------------------------------------------------
1: my-domain.me
-------------------------------------------------------------------------------
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1

這邊是詢問 Redirect, 是否要將 http(port:80) 重新導向到 https


Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for my-domain.me
Waiting for verification...
Cleaning up challenges
Deploying Certificate for my-domain.me to VirtualHost /etc/apache2/sites-enabled/000-default-le-ssl.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Enhancement redirect was already set.

搭啦~ 然後就 renew 成功啦


-------------------------------------------------------------------------------
Your existing certificate has been successfully renewed, and the new certificate
has been installed.

The new certificate covers the following domains:
https://my-domain.me

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=my-domain.me
-------------------------------------------------------------------------------

貼心小提示, renew 完成的憑證路徑, 還有到期日 2017-12-02

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/my-domain.me/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/my-domain.me/privkey.pem
   Your cert will expire on 2017-12-02. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot-auto
   again with the "certonly" option. To non-interactively renew *all*
   of your certificates, run "certbot-auto renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

CronJob 自動化


Random

java.util.Random 其實一點也不 Random, 因為屬於一種線性分佈,
線性不比離散, 所以就是有公式可以預測啦~

大神的論文在 這裡

https://stackoverflow.com/questions/11051205/difference-between-java-util-random-and-java-security-securerandom

The standard Oracle JDK 7 implementation uses what's called a Linear Congruential Generator to produce random values in java.util.Random.

Predictability of Linear Congruential Generators

Hugo Krawczyk wrote a pretty good paper about how these LCGs can be predicted ("How to predict congruential generators"). If you're lucky and interested, you may still find a free, downloadable version of it on the web. And there's plenty more research that clearly shows that you should never use an LCG for security-critical purposes. This also means that your random numbers are predictable right now, something you don't want for session IDs and the like.

隨機數安全議題

http://wps2015.org/drops/drops/%E8%81%8A%E4%B8%80%E8%81%8A%E9%9A%8F%E6%9C%BA%E6%95%B0%E5%AE%89%E5%85%A8.html

SecureRandom

較推薦的做法是採用 java.security.SecureRandom,


寫個簡單的 Faker


import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.util.stream.Collectors;
import java.util.stream.IntStream;

/**
 * Created by jerry on 2017/11/1.
 */
public class Faker {

    private SecureRandom random;
    
    public Faker() throws NoSuchAlgorithmException {
        this.random = SecureRandom.getInstance("SHA1PRNG");
    }
    
   /**
     * Generate a random alpha numbs.
     *
     * @param length
     * @return
     */
    public String randomAlphaNums(int length) {
        final char[] chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789".toCharArray();
        final String result = IntStream.range(0, length)
                .boxed()
                .map(index -> {
                    char character = chars[random.nextInt(chars.length)];
                    return String.valueOf(character);
                }).collect(Collectors.joining());

        return result;
    }

   /**
     * Generate a uuid.
     *
     * @return
     */
    public String uuid() {
        return UUID.randomUUID().toString();
    }

    /**
     * Generate a fake url
     *
     * @param domain
     * @return
     */
    public String randomUrl(String domain) {
        return new StringBuilder("https://")
                .append(domain)
                .append("/" + randomAlphaNums(5))
                .append("/" + randomAlphaNums(5))
                .toString();
    }
}


@Test
public void testAlphaNumbs() throws NoSuchAlgorithmException {
    Faker faker = new Faker();
    
    // mO03nYMKAiQi06sleVgeQSP4ZWpD5O2sr4M9PXyj6GBA0VHY2ucS9J0s4atRSRovI0EjffBoqBFL3loaLrbpKrxlkHFDHs76nMzW
    System.out.println(faker.randomAlphaNums(100));

    // https://my.domain/KMr1n/3xTIn
    System.out.println(faker.randomUrl("my.domain"));

    // 0b256f71-77bf-4dfb-8fae-850a128c785b
    System.out.println(faker.uuid());
}

題外話 Java Faker

Java Faker 是一款改寫自 Ruby's stympy/faker gem 的小工具, 在 TDD 階段還算滿實用的.




Chung Nguyen

Sample Data


[
    {"uid":"1","name":"阿花","age":38,"gender":"F"},
    {"uid":"2","name":"阿瓜","age":28,"gender":"M"}
]

Entity


@Data
@JsonInclude(JsonInclude.Include.NON_NULL)
@JsonPropertyOrder(value = {
    "uid",
    "name",
    "age",
    "gender"
})
public class Customer {
    // creativeId equals adId
    @JsonProperty("uid")
    private String uid;
    @JsonProperty("name")
    private String name;
    @JsonProperty("age")
    private Integer age;
    @JsonProperty("gender")
    private String gender;
}

Test


@Test
public void test() throws IOException {
    List<Customer> customerList =
        objectMapper.readValue(json, new TypeReference>() {});

    Map customerMap = customerList.stream()
        .collect(Collectors.toMap(
            Customer::getUid,       // key
            customer -> customer)); // value

    System.out.println(objectMapper.writeValueAsString(customerMap));
}

Result


{
  "1": {
    "uid": "1",
    "name": "阿花",
    "age": 38,
    "gender": "F"
  },
  "2": {
    "uid": "2",
    "name": "阿瓜",
    "age": 28,
    "gender": "M"
  }
}


Optional

Java 最大的敵人, 我想應該就是 NPT Exception, tears.
所以 Java8 設計了一個 Optional 用來解決 NPT 這淘氣的小東西, fuck.
詳細的使用方式

參閱這個 bloger 的介紹, 寫的超棒, 我覺得啦 :)


JpaRepository

Spring framework 做 CRUD 的好朋友, 就是 JpaRepository,
沒有用 Optional 的寫法像這樣


@Repository
public interface UserInfoRepository extends JpaRepository<UserInfo, String> {

    UserInfo findByName(String name);
}

這樣寫其實也沒什麼不合理, 只是在商業邏輯處理時, 要多一個 if 判斷,
判斷是否有查詢到 userInfo 的資料.


public void business() {

    UserInfo user = userInfoRepository.findByName("dumdum");

    if (Objects.isNull(user)) {
        // throw exception

    } else {
        // do something
    }

}

改為用 Optional 試試看


@Repository
public interface UserInfoRepository extends JpaRepository<UserInfo, String> {

    Optional<Userinfo> findByName(String name);
}


public void business() {

    Optional<Userinfo> userOpt = userInfoRepository.findByName("dumdum");
    UserInfo user = userOpt.orElseThrow(() ->
            new Exception("查無 userInfo(dumdum) 資訊."));

    // do something

}

這樣寫感覺好多了, 另一方面也可以清楚的明白 Exception 是什麼原因產生的

找不到 user 還可以替換為預設 user, 這樣就不用處理 Exception,
當然並不是所有的情境都可以預設一個 instance.


public void business() {

    Optional<Userinfo> userOpt = userInfoRepository.findByName("dumdum");
    UserInfo user = userOpt.orElse(new UserInfo("預設 user")));

    // do something

}

1. 產生 SSL certificate


keytool -genkey -alias tomcat -storetype PKCS12 -keyalg RSA -keysize 2048 -keystore keystore.p12 -validity 3650


Enter keystore password:
Re-enter new password:
What is your first and last name?
  [Unknown]:  jerry
What is the name of your organizational unit?
  [Unknown]:  td
What is the name of your organization?
  [Unknown]:  com
What is the name of your City or Locality?
  [Unknown]:  taipei
What is the name of your State or Province?
  [Unknown]:  taiwan
What is the two-letter country code for this unit?
  [Unknown]:  tw
Is CN=jarvis, OU=td, O=urad, L=taipei, ST=taiwan, C=tw correct?
  [no]:  yes

這個 certificate 是 self-signed certificate 沒有經過第三方認證, 所以沒有公信力,
正式上線會在瀏覽器看到 連線不被信任

要有公信力的 certificate 最簡單的是 Lets Encrypt, 其他就是花一些錢找簽發 certificate 的組織

2. Enable HTTPS in Spring Boot

Spring Boot 內建的 tomcat 預設 http 是 8080, Spring Boot 可以設定 http 跟 https,
但沒辦法同時存在這兩個設定, 如果要同時存在兩種 connection,
建議依照 文件 建議設定 https,
再透過 programmatically 去設定 http 會比較容易。

相關範例可以參考 : https://github.com/spring-projects/spring-boot/tree/master/spring-boot-samples/spring-boot-sample-tomcat-multi-connectors

application.properies 設定參考


server:
    port: 8443
    ssl.key-store: keystore.p12
    ssl.key-store-password: myKeyPassword
    ssl.keyStoreType: PKCS12
    ssl.keyAlias: tomcat

3. Redirect HTTP to HTTPS



import org.apache.catalina.Context;
import org.apache.catalina.connector.Connector;
import org.apache.tomcat.util.descriptor.web.SecurityCollection;
import org.apache.tomcat.util.descriptor.web.SecurityConstraint;
import org.springframework.boot.context.embedded.EmbeddedServletContainerFactory;
import org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;

/**
 * Created by jerry on 2017/10/12.
 * 重新導向 Http 到 Https (只運作於 prod 環境)
 * https://drissamri.be/blog/java/enable-https-in-spring-boot/
 */
@Profile("prod")
@Configuration
public class WebConfiguration {

    @Bean
    public EmbeddedServletContainerFactory servletContainer() {
        TomcatEmbeddedServletContainerFactory tomcat = new TomcatEmbeddedServletContainerFactory() {
            @Override
            protected void postProcessContext(Context context) {
                SecurityConstraint securityConstraint = new SecurityConstraint();
                securityConstraint.setUserConstraint("CONFIDENTIAL");
                SecurityCollection collection = new SecurityCollection();
                collection.addPattern("/*");
                securityConstraint.addCollection(collection);
                context.addConstraint(securityConstraint);
            }
        };

        tomcat.addAdditionalTomcatConnectors(initiateHttpConnector());
        return tomcat;
    }

    private Connector initiateHttpConnector() {
        Connector connector = new Connector("org.apache.coyote.http11.Http11NioProtocol");
        connector.setScheme("http");
        connector.setPort(8080);
        connector.setSecure(false);
        connector.setRedirectPort(8443);

        return connector;
    }
}

Entity



@Data
@NoArgsConstructor
@AllArgsConstructor
@Entity(name = "messenger")
@IdClass(MessengerPrimaryKey.class)
public class Messenger implements Serializable {
    @Id
    @Column(name = "user_id", length = 64, nullable = false)
    private String userId;
    @Id
    @Column(name = "psid", length = 64, nullable = false)
    private String psId;

    @Column(name = "page_id", length = 64, nullable = false)
    private String pageId;

}

PrimaryKey


@Data
@NoArgsConstructor
@AllArgsConstructor
public class MessengerPrimaryKey implements Serializable {
    private String userId;
    private String psId;
}

Test1

transactionalTest1 是基本的 transactional, 在 transactionalTest1 結束後會做一個 commit

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.transaction.annotation.Transactional;
import tw.com.urad.entities.Messenger;

import javax.inject.Inject;

@RunWith(SpringRunner.class)
@SpringBootTest
public class MessengerRepositoryTest {

    @Inject
    private MessengerRepository repository;

    @Test
    @Transactional
    public void transactionalTest1() {
        final String userId = "userId";
        final String psId = "psId";
        final String pageId = "pageId";
        Messenger messenger = new Messenger(userId, psId, pageId);

        Messenger saved = repository.save(messenger);
        
        // Messenger(userId=userId, psId=psId, pageId=pageId)
        System.out.println(saved);

        // modified saved entity
        saved.setPageId("pageId2");

        // [Messenger(userId=userId, psId=psId, pageId=pageId)]
        System.out.println(repository.findAll());
    }
}

Test2

翻了一下 文件 裡面有一段這樣的敘述:

The readOnly flag instead is propagated as hint to the underlying JDBC driver for performance optimizations. Furthermore, Spring will perform some optimizations on the underlying JPA provider. E.g. when used with Hibernate the flush mode is set to NEVER when you configure a transaction as readOnly which causes Hibernate to skip dirty checks (a noticeable improvement on large object trees).

當 transactional 是 readOnly 的狀態, 整個 transactions 基本上就只有 read 的動作可以執行, 對有做讀寫分離的資料庫是一種不錯的策略。然後還會幫你優化底層的 JDBC Driver 的處理效能, 怎麼優化呢? 是因為他會跳過 Hibernate 的 dirty checks, 在資料庫資料量龐大的時候特別顯著。

dirty checks 是為了避免一些 "異常" 的資料, 指的應該是 lost update, dirty read, phantom read 這些意外, 我猜這個 "dirty checks" 指的是 persistence 資料的正確性檢查!?

參考資料:http://www.importnew.com/12314.html

至於為何 System.out.println(saved); 仍會有資料被 print, 是因為 save 回傳的是 entity, 這個 entity 是當初要寫入資料庫的那個, 並非在 transactional 階段的資料。

Saves a given entity. Use the returned instance for further operations as the save operation might have changed the entity instance completely.

參考資料:https://docs.spring.io/spring-data/commons/docs/current/api/org/springframework/data/repository/CrudRepository.html


@RunWith(SpringRunner.class)
@SpringBootTest
public class MessengerRepositoryTest {

    @Inject
    private MessengerRepository repository;

    @Test
    @Transactional(readOnly = true)
    public void transactionalTest2() {
        final String userId = "userId";
        final String psId = "psId";
        final String pageId = "pageId";
        Messenger messenger = new Messenger(userId, psId, pageId);

        Messenger saved = repository.save(messenger);
        
        // Messenger(userId=userId, psId=psId, pageId=pageId)
        System.out.println(saved);

        // modified saved entity
        saved.setPageId("pageId2");

        // nothing
        System.out.println(repository.findAll());
    }
}


Test3

我在這個測試多增加了 @FixMethodOrder(MethodSorters.DEFAULT) 讓測試按照 alpha-number 的順序執行, 拿掉 readOnly, 理論上 void transactionalTest2() 應該會有資料, 但實際上並沒有。

If your test is @Transactional, it will rollback the transaction at the end of each test method by default. However, as using this arrangement with either RANDOM_PORT or DEFINED_PORT implicitly provides a real servlet environment, HTTP client and server will run in separate threads, thus separate transactions. Any transaction initiated on the server won’t rollback in this case.

參考資料:https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-testing.html#boot-features-testing-spring-boot-applications

就是說在 @SpringBootTest 的環境下, 為了避免弄髒真實的資料庫(拿真實的資料庫來做測試心臟也是很強大啊), 可以透過 @Transactional 在測試結束後 rollback。



@RunWith(SpringRunner.class)
@SpringBootTest
@FixMethodOrder(MethodSorters.DEFAULT)
public class MessengerRepositoryTest {

    @Inject
    private MessengerRepository repository;

    @Test
    @Transactional
    public void transactionalTest1() {
        final String userId = "userId";
        final String psId = "psId";
        final String pageId = "pageId";
        Messenger messenger = new Messenger(userId, psId, pageId);

        Messenger saved = repository.save(messenger);
        System.out.println("===");
        System.out.println(saved);

        saved.setPageId("pageId2");
        System.out.println("===");
        System.out.println(repository.findAll());
    }

    @Test
    @Transactional
    public void transactionalTest2() {
        // print nothing
        System.out.println(repository.findAll());
    }
}

Todd Quackenbush

Facebook Messenger Platform

最主要的還是看文件, 偶而需要使用 Facebook Search 功能, 可以看到神秘版的文件。

1. 粉絲頁 與 App 的關係

在開發 Bot 之前, 需要做幾件事情

  1. 申請 facebook app
  2. 申請 facebook fans-page
  3. 規劃流程, 最好有一個懂 UX 的開發者參與, 不然流程會很 XD


一個 bot 需要一個處理各種 events 的 callback,
一個 app 只能設定一個 callback, 所以 bot 跟 app 是 1 對 1 的關係,
一個 page 可以被最多 10 個 app 訂閱, 所以 page 跟 app 是 1 對 n(n<=10) 的關係,


這會有什麼影響呢? 在多人協作開發的時候, 你的好 partner 會送很多 event 給你,
禮尚往來, 你也會送一些 event 給他, 有幾個方式

  1. 從原生 App 申請一組開發 app 出來, webhook 只設定自己開發要用的 event
  2. 額外申請一個自己專用的 Fans-Page, 來 subscribed


2. GetStartButton Not Work

這邊的雷是因為, 自己開發的時候只 hook 了 messages 這個 event,
後來開發的時候, 就乾脆都拿, 之後再看用了哪些功能在慢慢推給 Facebook 審核

  1. check GetStartButton setting
    
    curl -X GET "https://graph.facebook.com/v2.10/me/messenger_profile?fields=get_started&access_token="
    
  2. check webhook messaging_postbacks event

  3. check your callback rsources is exist.

Nolan Issac

使用 JMH 进行微基准测试:不要猜,要测试!

這幾天讀了幾篇很有趣的文章, 是關於 lambda 跟 jvm 效能評估的文章,
分別是
Java8 Lambda表达式和流操作如何让你的代码变慢5倍
使用JMH进行微基准测试:不要猜,要测试!


JMH

JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targetting the JVM.

JMH 是用來衡量 JVM 容器上運作的
(Java, Scala, Kotlin, Groovy, Clojure, etc.) 效能的工具,
官方建議透過 maven 來建立測試專案, 可以避免一些奇怪設定影響效能的問題,
groupdId 就替換成自己的 package name 吧,
artifactId 就替換成測試的 project name, 會按照這個 project name 在當下路徑建立一個資料夾,


mvn archetype:generate \
          -DinteractiveMode=false \
          -DarchetypeGroupId=org.openjdk.jmh \
          -DarchetypeArtifactId=jmh-java-benchmark-archetype \
          -DgroupId=com.example \
          -DartifactId=jmh-benchmark \
          -Dversion=1.0

maven Builde 出來的 code

import org.openjdk.jmh.annotations.Benchmark;

public class MyBenchmark {

    @Benchmark
    public void testMethod() {
        // This is a demo/sample template for building your JMH benchmarks. Edit as needed.
        // Put your benchmark code here.
    }

}

不管 3721 跑了再說,


$ cd jmh-benchmark
$ mvn clean install
$ java -jar target/benchmarks.jar

運行結果


# JMH version: 1.19
# VM version: JDK 1.8.0_92, VM 25.92-b14
# VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/bin/java
# VM options: 
# Warmup: 20 iterations, 1 s each
# Measurement: 20 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: com.example.MyBenchmark.testMethod

...

# Run progress: 90.00% complete, ETA 00:00:40
# Fork: 10 of 10
# Warmup Iteration   1: 3189934685.250 ops/s
# Warmup Iteration   2: 3068794073.424 ops/s
# Warmup Iteration   3: 3236114041.508 ops/s
# Warmup Iteration   4: 3134197602.346 ops/s
# Warmup Iteration   5: 3148638128.516 ops/s
# Warmup Iteration   6: 3173960761.557 ops/s
# Warmup Iteration   7: 3169265377.660 ops/s
# Warmup Iteration   8: 3077919609.443 ops/s
# Warmup Iteration   9: 3166464153.044 ops/s
# Warmup Iteration  10: 3078477796.372 ops/s
# Warmup Iteration  11: 3099929982.724 ops/s
# Warmup Iteration  12: 3036217732.999 ops/s
# Warmup Iteration  13: 3006113218.527 ops/s
# Warmup Iteration  14: 3181689542.757 ops/s
# Warmup Iteration  15: 3119538331.842 ops/s
# Warmup Iteration  16: 3105301506.039 ops/s
# Warmup Iteration  17: 3048849508.645 ops/s
# Warmup Iteration  18: 3072101635.390 ops/s
# Warmup Iteration  19: 3144030955.986 ops/s
# Warmup Iteration  20: 3151332956.538 ops/s
Iteration   1: 3164452829.947 ops/s
Iteration   2: 3113814464.802 ops/s
Iteration   3: 3052097465.786 ops/s
Iteration   4: 2980582092.738 ops/s
Iteration   5: 3042852421.053 ops/s
Iteration   6: 3091791507.999 ops/s
Iteration   7: 3137421146.240 ops/s
Iteration   8: 2921380582.360 ops/s
Iteration   9: 2991112676.148 ops/s
Iteration  10: 3141795516.024 ops/s
Iteration  11: 3106155855.084 ops/s
Iteration  12: 3099071004.994 ops/s
Iteration  13: 3165802127.937 ops/s
Iteration  14: 3117700695.037 ops/s
Iteration  15: 3184560049.227 ops/s
Iteration  16: 3176507506.414 ops/s
Iteration  17: 3133869785.413 ops/s
Iteration  18: 3150455744.882 ops/s
Iteration  19: 3189445594.500 ops/s
Iteration  20: 3089706559.098 ops/s


Result "com.example.MyBenchmark.testMethod":
  3133357710.097 ±(99.9%) 12882845.404 ops/s [Average]
  (min, avg, max) = (2921380582.360, 3133357710.097, 3284444409.646), stdev = 54546774.121
  CI (99.9%): [3120474864.693, 3146240555.501] (assumes normal distribution)


# Run complete. Total time: 00:06:44

Benchmark                Mode  Cnt           Score          Error  Units
MyBenchmark.testMethod  thrpt  200  3133357710.097 ± 12882845.404  ops/s

說明

我沒有找到很明確的說明文件, 下面的說明是從網路上整理來跟一部份自己猜測的

  • 運行的時候應盡量關閉不必要的 applications , 確保沒有其他變因。
  • 運行的次數越多越好, 避免 max / min 影響, 結果是取平均值。
  • 最上方的說明, Warmup: 20 iterations, 1 s each 預熱 20 次, 每次執行 1s, 不是很確定預熱的原理跟用意, 猜測是減少啟動 JVM 所造成的變因?
  • 最上方的說明, Measurement: 20 iterations, 1 s each 評估 20 次, 每次執行 1s。
  • 評估的單位是 ops/s (operations per second), 每秒運行有 @Benchmark 標記的 method 的次數。
  • Benchmark mode: Throughput, ops/time Throughput (吞吐量), 每秒可以運作的次數作為衡量標準, 其他 mode 下面補充。
  • 最下方的 Result, 3133357710.097 ±(99.9%) 12882845.404 ops/s, 平均是 3133357710.097 ops/s , 誤差上下 12882845.404 ops/s。
  • (min, avg, max) = (2921380582.360, 3133357710.097, 3284444409.646), stdev = 54546774.121, stdev 樣本標準差 54546774.121。
  • CI (99.9%): [3120474864.693, 3146240555.501], 常態分佈(高斯分佈) !? 數學不好, 不太確定。

Mode

  • Mode.Throughput - 評估時間內吞吐量, 單位時間內的執行次數。
  • Mode.AverageTime - 評估執行平均時間, 多次執行花費的時間, 取平均值。
  • Mode.SampleTime - 評估樣本(Sample) 的執行時間, (n %) Sample 在某個時間內執行完成
  • Mode.SingleShotTime - 冷測試評估, 不做 JVM warm up, 只執行一次有 @Benchmark 標記的 method, 用來評估在 JVM 啟動運行所需的時間。

Dead Code Elimination

Dead Code 指的是沒有被使用的 code, 比如下方的 int sum = a + b;,
sum 沒有繼續做任何運算處理, jmh 會因為 sum 沒有被使用,
而忽略評估 a + b 這段運算, 所以評估就不準啦。


    @Benchmark @BenchmarkMode(Mode.Throughput)
    public void testMethod() {
        // This is a demo/sample template for building your JMH benchmarks. Edit as needed.
        // Put your benchmark code here.

        int a = 1;
        int b = 2;
        int sum = a + b;
    }

Avoiding Dead Code Elimination

解決 Dead Code 的方法,

  1. return sum, 讓 sum 確實有被使用。
  2. Passing Value to a Black Hole , 意思是弄一個黑洞把變數丟進去, 假裝黑洞用了那個變數, 類似 Mockito 的 @Mock, 做法看下來

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.infra.Blackhole;

public class MyBenchmark {

    @Benchmark
    public void testMethod(Blackhole blackhole) {
        int a = 1;
        int b = 2;
        int sum = a + b;
        blackhole.consume(sum);
    }
}

Return the result of your code from the benchmark method.
Pass the calculated value into a "black hole" provided by JMH.

大大們的筆記

  1. tutorials.jenkov.com, 滿詳細的一篇, http://tutorials.jenkov.com/java-performance/jmh.html#your-first-jmh-benchmark
  2. java-performance.info - http://java-performance.info/jmh/
  3. importnew - http://www.importnew.com/12548.html
  4. blog.dyngr.com - http://blog.dyngr.com/blog/2016/10/29/introduction-of-jmh/

題目

Merge two sorted linked lists and return it as a new list.
The new list should be made by splicing together the nodes of the first two lists.

看完題目, 以為可以用 LinkedList 這類的東西, 想說 ez ...
結果下面給的 hint Sample 根本就不是那麼一回事啊, 抓頭抓抓抓...


/**
 * Definition for singly-linked list.
 *
 */
public class ListNode {
    int val;
    ListNode next;
    ListNode(int x) { val = x; }
}

先拿到測試資料


/**
 * Created by jerry on 2017/9/24.
 *
 * Merge two sorted linked lists and return it as a new list.
 * The new list should be made by splicing together the nodes of the first two lists.
 */
public class LeetCode21MergeTwoSortedListsTest {

    private LeetCode21MergeTwoSortedLists sol = new LeetCode21MergeTwoSortedLists();

    @Test
    public void test1() {
        final ListNode l1 = null;
        final ListNode l2 = new ListNode(0);

        ListNode act = sol.mergeTwoLists(l1, l2);

        Assert.assertEquals(0, act.val);
    }

    @Test
    public void test2() {
        final ListNode l1 = new ListNode(2);
        final ListNode l2 = new ListNode(1);

        ListNode act = sol.mergeTwoLists(l1, l2);

        Assert.assertEquals(1, act.val);
        Assert.assertEquals(2, act.next.val);
    }

    @Test
    public void test3() {
        final ListNode l1 = new ListNode(1);
        final ListNode l2 = new ListNode(2);

        ListNode act = sol.mergeTwoLists(l1, l2);

        Assert.assertEquals(1, act.val);
        Assert.assertEquals(2, act.next.val);
    }

    @Test
    public void test4() {
        final ListNode l1 = new ListNode(5);
        final ListNode l2 = new ListNode(1);
        final ListNode l3 = new ListNode(2);
        final ListNode l4 = new ListNode(4);
        l2.next = l3;
        l3.next = l4;

        ListNode act = sol.mergeTwoLists(l1, l2);
        Assert.assertEquals(1, act.val);
        Assert.assertEquals(2, act.next.val);
        Assert.assertEquals(4, act.next.next.val);
        Assert.assertEquals(5, act.next.next.next.val);
    }

    @Test
    public void test5() {
        final ListNode l1 = new ListNode(-9);
        final ListNode l2 = new ListNode(3);
        l1.next = l2;

        final ListNode l3 = new ListNode(5);
        final ListNode l4 = new ListNode(7);
        l3.next = l4;

        ListNode act = sol.mergeTwoLists(l1, l3);

        Assert.assertEquals(-9, act.val);
        Assert.assertEquals(3, act.next.val);
        Assert.assertEquals(5, act.next.next.val);
        Assert.assertEquals(7, act.next.next.next.val);
    }
}

Solution

還是抓不太到這種遞迴的解法, 有種輾轉互相比較的意味


public class LeetCode21MergeTwoSortedLists {
    // [-9, 3], [5, 7]
    public ListNode mergeTwoLists(ListNode l1, ListNode l2) {
        if (Objects.isNull(l1)) return l2;
        if (Objects.isNull(l2)) return l1;

        if (l1.val < l2.val) {
            l1.next = mergeTwoLists(l2, l1.next);
            return l1;

        } else {
            l2.next = mergeTwoLists(l1, l2.next);
            return l2;
        }
    }
}

建立測試案例


public class LeetCode1TwoSumTest {

    final LeetCode1TwoSum twoSum = new LeetCode1TwoSum();

    /**
     * Example:
     * Given nums = [2, 7, 11, 15], target = 9,
     * Because nums[0] + nums[1] = 2 + 7 = 9,
     * return [0, 1].
     */
    @Test
    public void test1() {
        final int[] nums = new int[]{2, 7, 11, 15};
        final int target = 9;

        int[] act = twoSum.twoSum(nums, target);

        Assert.assertTrue(IntStream.of(act).anyMatch(x -> x == 0));
        Assert.assertTrue(IntStream.of(act).anyMatch(x -> x == 1));
    }

    @Test
    public void test2() {
        final int[] nums = new int[]{3, 3};
        final int target = 6;

        int[] act = twoSum.twoSum(nums, target);

        Assert.assertTrue(IntStream.of(act).anyMatch(x -> x == 0));
        Assert.assertTrue(IntStream.of(act).anyMatch(x -> x == 1));
    }

    @Test
    public void test3() {
        final int[] nums = new int[]{3, 2, 4};
        final int target = 6;

        int[] act = twoSum.twoSum(nums, target);

        Assert.assertTrue(IntStream.of(act).anyMatch(x -> x == 1));
        Assert.assertTrue(IntStream.of(act).anyMatch(x -> x == 2));
    }
}


Solution


public class LeetCode1TwoSum {

    public int[] twoSum(final int[] nums, final int target) {
        // 用來存放 target - n , HashMap
        HashMap map = new HashMap<>();
        int[] result = new int[2];

        for (int i = 0; i < nums.length; i++) {
            // 如果 map 有(target - n) 的值, 依序將 index 存入 result, 回傳 result
            if (map.containsKey(nums[i])) {
                int index = map.get(nums[i]);
                result[0] = index;
                result[1] = i;
                return result;
            } else {
                // 計算目前 index , 相差(target - n) 才會等於 target
                map.put(target - nums[i], i);
            }
        }
        return result;
    }
}

  1. 先隨便建立好, DataSet 跟 Table,
  2. 然後先手動匯入一筆 Row data,
  3. 安裝好 bigQuery CLI 工具

[Ubuntu 環境]

bq show --format=prettyjson DataSet.Table

如何实现团队的自组织管理
http://ifeve.com/self-organizing/

DynamoDB

Table

  • 類似於 RDBMS 的 Table.
  • DynamoDB Table 是一個儲存集合單位。
  • 相當於 MongoDB 的 Collection

Items

  • 每個 Table 可以有多個 Items,相當於 RDBMS 的 Rows。
  • 每個 Items 可包含多個 Attributes
  • 相當於 MongoDB 的 Document

Attributes:

  • 每個 Items 由一個或多個 Attributes 組成
  • Attributes 支援最深 32 個層級

Example

  • People Table
{
    "PersonID":"101",
    "LastName":"Simtih",
    "FirstName":"Fred",
    "Phone":"123-456",
},
{
    "PersonID":"102",
    "LastName":"Jones",
    "FirstName":"Anytown",
    "Address":{
        "Street":"123 Main",
        "City":"Anytown",
        "Zip":123
    }
}

Primary Key

Partitiion Key

  • 相當於 RDS 的 Unique Key, 有一組不會重複的 hash value

Partition Key + Sort Key (複合鍵)

  • 先找到 unique key 再依照 sort key 做索引, hash + range
  • Hash : select * from xxx where id = hash
  • Hash + range : select * from xxx where id between (123, 456)

Secondary Indexes (二級索引)

  • 二級索引的會有一張 base table 跟 關聯的 table
  • Global secondary index:
    1. Key = Hash Key or (Hash and Rang Key), 前面的 Hash Key 是 base Table 的 hash Key, 後面的 Hash and Range Key 是 join 的 Table
  • Local secondary index:
    1. Key = Hash Key and Range Key
  • 要注意的是,DynamoDB 不管是 Primary Key or Secondary Indexes,在 Table 建立之後就無法修改。

GCP Storage 簡介

Overview of storage classes

1.Multi-Regional Storage

適合存放經常存取的 "Hot Data" 或者影音串流(streaming videos),有異地備份的功能。

2.Regional Storage

適合存放與 Google Compute Engine Instance 或 Google Cloud DataProc 服務互相配合的資料,在相同的 Regional 可以降低 latency。Google Compute Engine 跟 Google Cloud DataProc 這些 instances 都是拿來處理, 清理, 運算資料用, 要分析的資料適合用這種類型的儲存。存放的費用比 Multi-Regional 還要低一點($0.016 GB/Month)。

3.Nearline Storage

適合存放較低存取的 "Cold Data",比如每月(隔 30 天才會存取一次)需要存取或修改一次的資料,或者適用於一些資料備份(用來回復跟救援的資料),支援整合 AWS Glacier 轉移。

4.Coldline Storage

適合存放較低存取的 "Cold Data",比起 Nearline Storage,Coldline Storage 更適合線上備份跟救援回復的資料(隔 90 天才會存取一次)。

Pricing

https://cloud.google.com/storage/pricing

Standar Storage 的建議

如果不清楚,資料的存取用途,建議使用 Standard Storage buckets

losf

lsof -n -i4TCP:8080 

Result

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
pc 13911 pc 133u IPv6 0xxxxxxxxxxxxxxxxx 0t0 TCP *:http-alt (LISTEN)

JQ tutorial

Sample

$docker network inspect bridge

[{
  "Name": "bridge",
  "Id": ""fake-id",
  "Created": "2017-07-05T01:07:55.747497179Z",
  "Scope": "local",
  "Driver": "bridge",
  "EnableIPv6": false,
  "IPAM": {
    "Driver": "default",
    "Options": null,
    "Config": [{
      "Subnet": "172.17.0.0/16",
      "Gateway": "172.17.0.1"
    }]
  },
  "Internal": false,
  "Attachable": false,
  "Ingress": false,
  "ConfigFrom": {
    "Network": ""
  },
  "ConfigOnly": false,
  "Containers": {
    "b59a12af7c87621b43c95a9adbcf047076b8a4b6b6ce0d6c5e6e48e21188ce6a": {
    "Name": "heuristic_rosalind",
    "EndpointID": "fake-endpoint-id",
    "MacAddress": ""fake-mac-address",
    "IPv4Address": "172.17.0.2/16",
    "IPv6Address": ""
  }},
  "Options": {
  "com.docker.network.bridge.default_bridge": "true",
  "com.docker.network.bridge.enable_icc": "true",
  "com.docker.network.bridge.enable_ip_masquerade": "true",
  "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
  "com.docker.network.bridge.name": "docker0",
  "com.docker.network.driver.mtu": "1500"
  },
  "Labels": {}
}]

寫個 shell 自動取得 IPv4Address

#!/bin/bash
#先拿到 `Containers` 底下的那串 hash
containers=`docker network inspect bridgejq .[0].Containersjq 'keys[]'`
echo "====== 取得 Container instance :${containers[0]}"
#取得 ipv4 Address
ipv4=`docker network inspect bridge"jq .[0].Containers.${containers}.IPv4Address`

這樣就很容易建立 bridge 讓 containers 互連啦 ~~

Retrofit2 + OKHttp3 真是神器

很久之前寫了剛開始寫 Java ,整理了一下自己在 spring 上使用 Retrofit2 與 okHttp3 的使用 心得

最近又發現 OKHttp3 plugin MockWebServer。在做第三方 API 測試的時候,真的滿方便的。


MockWebServer

使用 Retrofit 建立一個 FacebookService,
建立方式, 參考之前的心得


public interface FacebookService {
    @GET("{userId}?fields=id, name")
    Call getFbUser(@Path("userId") String userId, @Query("access_token") String token);
}

測試


@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = MockWebServerDemo.class)
@WebAppConfiguration
public class MockWebServerTest {

    private MockWebServer mockWebServer;
    private FacebookService facebookService;
    private ObjectMapper objectMapper;

    @Before
    public void setUp() {
        // Jackson objectMapper
        objectMapper = new ObjectMapper();
        objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        objectMapper
            .registerModule(new JodaModule())
            .configure(DeserializationFeature.READ_DATE_TIMESTAMPS_AS_NANOSECONDS, false);

        // mockWebServer
        mockWebServer = new MockWebServer();

        // 注入到 Retrofit
        Retrofit mockRetrofit = new Retrofit.Builder()
            .baseUrl(mockWebServer.url(""))  // 給個空的 url
            .addConverterFactory(JacksonConverterFactory.create(objectMapper))
            .build();

        facebookService = mockRetrofit.create(FacebookService.class);
    }

    @After
    public void tearDown() throws IOException {
        // 記得 shutdown
        mockWebServer.shutdown();
    }

    @Test
    public void testFacebookService() throws IOException {
        final Integer testUserId = 123;
        final String testName = "fake-facebook-name";

        // 假裝 Facebook API Response 200
        mockWebServer.enqueue(new MockResponse()
            .setResponseCode(200)
            .setBody("{" +
                "  \"id\": \"" + testUserId + "\"," +
                "  \"name\": \"" + testName + "\"" +
                "}")
        );

        // act
        FacebookUserInfoDTO fbUser =
            facebookService.getFbUser(String.valueOf(testUserId), testName)
            .execute()
            .body();

        Assert.assertEquals(testUserId, fbUser.getId());
        Assert.assertEquals(testName, fbUser.getName());
    }

英文很差,所以附上原文。

關係大概是這樣吧

image-55

Processes And Threads

Processes (進程)

每個 Process 都可視為一個獨立運作的環境,有自己完整、私有的 Run-Time 資源,特別是每個 process 都有自己的記憶體空間。

Processes 聽起來很像 programs 或 application 的另一種說法,然而事實上一個 application 有可能是許多 processes 合作組成的。為了讓 processes 彼此互相合作,一些作業系統支援了 Inter Process Communication (IPC) resources,例如 pipes 與 sockets。IPC 不只讓不同的 processes 互相通訊,同時也支援不同系統的 pocesses 互相通訊。

大部分 JVM 實現的都是 single process。Java Application 可以創建額外的 process 透過 ProcessBuilder Object。

A process has a self-contained execution environment. A process generally has a complete, private set of basic run-time resources; in particular, each process has its own memory space.
Processes are often seen as synonymous with programs or applications. However, what the user sees as a single application may in fact be a set of cooperating processes. To facilitate communication between processes, most operating systems support Inter Process Communication (IPC) resources, such as pipes and sockets. IPC is used not just for communication between processes on the same system, but processes on different systems.
Most implementations of the Java virtual machine run as a single process. A Java application can create additional processes using a ProcessBuilder object. Multiprocess applications are beyond the scope of this lesson.

Threads (線程)

Thread 在某些時候被視為一種輕量的 processes (lightweight processes),與 processes 一樣,threads 也提供了一個可執行的環境。但相對於 processes 創建時,threads 並不需要太多的資源。

每個 process 最少都有一個 thread,threads 共享 processs 上的資源,包含 memory 與 files。這樣使得效率提高,但相對的也產生了一些潛在的問題,比如說 Race-condition 之類的問題。

Multithreaded 是 Java 平台上重要的一個特徵,每個 application 最少會有一個 thread。如果你去計算 "System" 上的 threads ,他們做的事情相似於記憶體管理或一些信號處理。但在這裡,我們只需要先認識 Main Thread,因為它能夠去創建額外的 threads。

Threads are sometimes called lightweight processes. Both processes and threads provide an execution environment, but creating a new thread requires fewer resources than creating a new process.
Threads exist within a process — every process has at least one. Threads share the process's resources, including memory and open files. This makes for efficient, but potentially problematic, communication.
Multithreaded execution is an essential feature of the Java platform. Every application has at least one thread — or several, if you count "system" threads that do things like memory management and signal handling. But from the application programmer's point of view, you start with just one thread, called the main thread. This thread has the ability to create additional threads, as we'll demonstrate in the next section.

使用時機

  1. IO bound task,避免 IO blocking 的阻塞。
  2. CPU bound task,避免長時間大運算的邏輯的阻塞,藉由 threads 的使用,有效率地利用 CPU 。
  3. Schedule,ScheduledThreadPoolExecutor 會用到 threads
  4. Daemon,daemon 是一種在 backgrouopd 運作的 thread,當所有的 non-daemon 都結束了,daemon 就會自動終止。另外從 daemon 裡面建立出來的 thread 也都會歸類在 daemon-thread。

Life cycle

image-7

圖片來自:https://github.com/JustinSDK/JavaSE6Tutorial/blob/master/docs/CH15.md#1514-執行緒生命周期

Synchonized 同步

public synchronized void myMethod() {
    //code
}

synchonized 用來保證一次只會有一個 thread 對某個資源的存取權,多執行緒對相同的 Object 做存取時,會發生:

DeadLock

比較經典的例子,哲學家吃飯
https://www.tutorialspoint.com/java/java_thread_deadlock.htm

Race Condition

比較經典的例子,應該就是提款機、賣票。
A 帳戶同時間發生, user1 與 user2 匯款,如果沒有 synchonized 的處理,會有

  1. user1 與 user2 都匯款成功, save
  2. user1 與 user2 都匯款成功, user2 的匯款資料蓋過 user1 , 損失 user1 的匯款金額。
  3. user1 與 user2 都匯款成功, user1 的匯款資料蓋過 user2 , 損失 user2 的匯款金額。

Heroku API 設計指南

  • 總是用 TLS
  • 使用 ETag
  • 提供 created_atupdated_at 時間戳記
  • 時間使用 ISO8601 UTC 格式,2012-01-01T12:00:00Z
  • 使用 Leaky Bucket 來限制 Request Rate Limit,這個水桶理論 滿有趣的
  • Response 壓縮 JSON ,去掉空白,換行符號
  • 產生 JSON 文件,並提供呼叫範例 PRMD
  • 描述 Resource 的穩定性

Bucket 原理

參考:https://zhuanlan.zhihu.com/p/20872901

  1. 所有的流量在放行之前需要獲取一定量的token;
  2. 所有的token 存放在一個bucket(桶)當中,每1/r 秒,都會往這個bucket 當中加入一個token;
  3. bucket 有最大容量(capacity or limit),在bucket 中的token 數量等於最大容量,而且沒有token 消耗時,新的額外的token 會被拋棄。

程序员职业生涯巡礼

  1. 程序员是个好职业
  2. 程序员是一个具备长久生命力的职业
  3. 程序员不一定要写一辈子程序
  4. StuQ 程序员技能 图谱
  5. 你不是一个人在编程
  6. 专业性很重要,但也别太「专」了
  7. 没什么职业规划,往前走,就是规划

Prism

image-2

Prism 是一款簡單又好看,好用的要死的 Code Syntax Highlighting 套件,輕量化,而且 plugin 又多方便套用自己擅長的的程式語言。

使用方式

安裝在 Blogger 上其實大同小異,可以參考

免費的 CDN

一開始我是用 GoogleDriver 來當免費的 css 跟 js hosting,但幾次改版後,hosting 的功能就失效了。於是改用免費的 CDN 。
推薦一下 JSDELIVR,可以免費幫你把多個 js ,Collection 成一個 resource,一次引用
http://www.jsdelivr.com/projects/prism

這個 Bloger 用到的 Resource

css

https://cdn.jsdelivr.net/g/prism@1.6.0(plugins/ie8/prism-ie8.css+themes/prism-okaidia.css+plugins/line-numbers/prism-line-numbers.css+plugins/previewer-color/prism-previewer-color.css)

css 使用方式


<link href='https://cdn.jsdelivr.net/g/prism@1.6.0(plugins/ie8/prism-ie8.css+themes/prism-okaidia.css+plugins/line-numbers/prism-line-numbers.css+plugins/previewer-color/prism-previewer-color.css)' rel='stylesheet'/>

JS

https://cdn.jsdelivr.net/g/prism@1.6.0(prism.js+components/prism-java.min.js+components/prism-javascript.min.js+components/prism-php-extras.min.js+components/prism-php.min.js+components/prism-bash.min.js+components/prism-clike.min.js+components/prism-markup.min.js+plugins/line-numbers/prism-line-numbers.min.js+plugins/previewer-color/prism-previewer-color.min.js)

JS 使用方式


<script src='https://cdn.jsdelivr.net/g/prism@1.6.0(prism.js+components/prism-java.min.js+components/prism-javascript.min.js+components/prism-php-extras.min.js+components/prism-php.min.js+components/prism-bash.min.js+components/prism-clike.min.js+components/prism-markup.min.js+plugins/line-numbers/prism-line-numbers.min.js+plugins/previewer-color/prism-previewer-color.min.js)' type='text/javascript'/>

給我資料

最近試著在蒐集一些看起來有點用途的資料,Google Trends 看起來是一個 “滿有價值” 的資料,是 Google 蒐集的全球趨勢資料,包含最近流行的關鍵字,關鍵字與關鍵字的比較,etc.,但 Google 並沒有提供相關的 API 介面,所以要取得資料的做法我第一個想到的是 Crawler,於是順手找了一下大神的開源專案,左看右看上看下看,都沒認真看,都覺得不順手XD。

Hack Fun

剛好讀到一篇很有趣的文章-hacking the google trends api,文章內容提到了幾個有趣的 end-point,像這樣

http://www.google.com/trends/fetchComponent?hl=en-US&q=html5,jquery&cid=TIMESERIES_GRAPH_0&export=5&w=500&h=300

URI http://www.google.com/trends/fetchComponent
hl en-US
q keywords
cid TIMESERIES_GRAPH_0/RISING_QUERIES_1_0/business_and_politics
export 5
w width
h height

還有這樣 curl 大絕招

curl --data "ajax=1&htd=20131111&pn=p1&htv=l" http://www.google.com/trends/hottrends/hotItems

還有直接把這個貼在 Browser

http://hawttrends.appspot.com/api/terms/

這邊還有大神整理的參數對照表,實在棒棒der。
http://myweb.fcu.edu.tw/~mhsung/Research/InformationSystem/JSON/JSON_24.htm

Browser Dev-Tools

看完大神的示範後,發現其實有些開放的資料並不一定要特別透過 Crawler 去取得,太沒效率了。而且大部份的瀏覽器都有提供開發工具,Hacking 那些 end-point 其實滿簡單的,比如說 Chrome 的開發者工具,我以這個潮潮的頁面來做示範好了

https://www.google.com/trends/hottrends/visualize?pn=p1

image-4

就可以愉快得到一串 curl

curl 'https://www.google.com/trends/hottrends/visualize/internal/data' -H 'Referer: https://www.google.com/trends/hottrends/visualize?pn=p1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36' --compressed

就可以愉快地寫 api 了

Sample

Reference

Required Java 1.8 Above

Play-Framework 大多看到的教學都是以 scala 為主,需要 Java 1.8 以上的版本。

SBT

SBT 跟 Activator 只要選擇一種方式安裝即可,兩種都算是 CI 統一開發環境的工具。

Activator

Activator 的好處就是有美美的 UI,
常用的指令

activator ui
activator list-templates

安裝 play

官網介紹了兩種方式

  1. Install Play With SBT
  2. Install Play With Activator

Data Source

private static List> teams = new ArrayList<>();
    private static List aTeam = new ArrayList<>();
    private static List bTeam = new ArrayList<>();
    static {
        aTeam.add("Java");
        aTeam.add("C++");
        aTeam.add("PHP");
        aTeam.add("Scala");

        bTeam.add("Go");
        bTeam.add("R");

        teams.add(aTeam);
        teams.add(bTeam);
    }

MapDemo

@Test
    public void mapDemo() {
        System.out.println("=== Demo Lambda Map ===");
        List> result1 = teams.stream().map(team -> {
            System.out.println("Team: " + team);
            return team;
        }).collect(Collectors.toList());

        System.out.println("Result: " + result1);
    }
=== Demo Lambda Map ===
Team: [Java, C++, PHP, Scala]
Team: [Go, R]
Result: [[Java, C++, PHP, Scala], [Go, R]]

FlatMap Demo

@Test
    public void flatMapDemo() {
        System.out.println("=== Demo Lambda FlatMap ===");
        List result2 = teams.stream().flatMap(team -> {
            System.out.println("Team: " + team);
            return team.stream();
        }).collect(Collectors.toList());

        System.out.println("Result: " + result2);
    }
=== Demo Lambda FlatMap ===
Team: [Java, C++, PHP, Scala]
Team: [Go, R]
Result: [Java, C++, PHP, Scala, Go, R]

Sample

Ec2 Instance(Consumer)
|
+- Worker +
|- Processor <-- shard (shard 的 seqId, partictionId 會被記錄到 DynamoDB)
|- Processor <-- shard (shard 的 seqId, partictionId 會被記錄到 DynamoDB)
|- Processor <-- shard (shard 的 seqId, partictionId 會被記錄到 DynamoDB)
|- Processor <-- shard (shard 的 seqId, partictionId 會被記錄到 DynamoDB)

Ec2 Instance(Consumer)
|
+- Worker +
|- Processor <-- shard (shard 的 seqId, partictionId 會被記錄到 DynamoDB)
|- Processor <-- shard (shard 的 seqId, partictionId 會被記錄到 DynamoDB)
|- Processor <-- shard (shard 的 seqId, partictionId 會被記錄到 DynamoDB)
|- Processor <-- shard (shard 的 seqId, partictionId 會被記錄到 DynamoDB)

Concept

每一台 Ec2 可視為一個 Consumer,每個 Consumer 會有個 Worker 機制把 shard Stream 分配給 Processor,當 shard 的數量調大的時候,KCL(Kinesis Clinet Libary) 裡面的 IRecordProcessorFactory 就會自動建立對應, shard 的 processor。

當有新的 Consumer 被建立時,AWS 會自動做 loadbalance,去平衡每個 worker 要負責的 shards,這時候 Processor 裡面的 shutdown 會被執行。

切記 Consumer 不應該開的比 shard 的數量還大。

1. FP, Functional Programming

不是有 function 的程式語言就叫做 functional programming language 唷?
https://ihower.tw/blog/archives/6305
讀完, 還是很難體會 FP。

* FP 精髓

http://www.codedata.com.tw/social-coding/paradigm-shift-to-functional-programming

  • map - 尋訪每個元素,加以處理,並且回傳處理後的元素。
  • filter - 回傳 布林值,以決定是否處理該元素。
  • reduce - 尋訪每個元素,依序組合元素,轉換成結果,丟給下個元素運算組合,然後產生最終組合的結果。

* Lazy

Call by Need

2. FRP, Functional Reactive Programming

http://www.ithome.com.tw/voice/91328

Reactive 的重點在於辨識出資料流,例如可以在欄位C1輸入=B1+5,然後在欄位D1輸入=C1+10,
此時,B1可以視為C1的資料來源,C1又可視為D1的資料來源,
每個欄位可以與其他欄位自由組合,形成資料流延續下去。
Reactive 是其目的,也就是強調必須即時地反應變化,非同步是達到此目的之手段,
為了能讓客戶端訂閱感興趣的資料流,採用了觀察者模式,
為了能讓開發者不落入如何處理(事件)資料的繁雜程式邏輯中,
採用了函數式的典範,隱藏了(事件)資料的迭代、轉換等細節,
從而能讓開發者根據規格進行宣告,以突顯出程式本身的意圖。

Taylor Durrer

Line-bot-sdk-java

前一陣子都在研究 line-bot-sdk-java,
看到 Retrofit2 搭配 OKHttp3,
媽媽就問我為什麼跪著寫程式。

Simsimi

Simsimi 聽說背後是百度的 NLP 技術的東西, 啊災...
因為我接完後, 他都回我幹話跟髒話, 我就決定叫他髒小雞,
他會根據一些關鍵字或什麼 NLP 之類的 (太難了) 來回我話, 支援多國語言,
對剛接觸 bot 來說, 算是滿方便, 又不用傷腦力去處理 NLP 的好服務, 當然你送的資料有可能都送給百度啦, maybe

Spring boot Framework

Spring boot framework 算是對新手滿友善的框架?那文件的厚實度, 印出來當枕頭都沒問題。
俗話說, 在哪裡跌倒, 就在哪裡躺下, 就在哪裡爬起來, Bean 很好用, 但有時候找不到文件就很雞掰。
Line-bot-sdk-java 的架構下了解一下 java 的注入, 營養真是滿分啊。

Purpose

包裝完後可以像這樣簡單俐落的使用。


Response simSimiResponse = simSimiService.chat("Hi~").execute();

RESTClient Service

Retrofit2 的起手式就是先墊一個 interface,透過 annonation 就可以很清楚的明暸 Resource,
滿符合 RESTful 的精神的啊, 在閱讀跟維護上也是好棒棒。
詳細的Retrofit2 Doc 請參閱這邊 https://square.github.io/retrofit/


public interface SimSimiService {
    /**
     * @param msg
     * @return ResponseBody {@link ResponseBody}
     * @see http://developer.simsimi.com/api
     */
    @GET("request.p")
    Call chat(@Query("text") String msg);
}

Implement Interface

line-idk-java 採用了 Builder Design Pattern, 不太確定是不是這樣稱呼?
因為 RESTClient 會有固定定的 Host,固定 Authorized token 跟一些 Header 欄位,
所以在組裝 HTTPClient 的時候採用 Builder 的方式比較容易,
比如說可以當 Api Url 更新時, 可以透過 builder 從外部注入。

Builder Design Pattern 範例 - https://sourcemaking.com/design_patterns/builder/java/2

Builder Design Pattern 說明 - http://design-patterns.readthedocs.io/zh_CN/latest/creational_patterns/builder.html


/**
 * SimSimiServiceBuilder
 * Builder with OkHttpClient Builder and Retrofit Builder
 */
public class SimSimiServiceBuilder {

    // 預設 api endpoint url
    public static final String DEFAULT_API_END_POINT = "http://sandbox.api.simsimi.com";

    // http 預設 connection 常數
    public static final long DEFAULT_CONNECT_TIMEOUT = 10_000;
    public static final long DEFAULT_READ_TIMEOUT = 10_000;
    public static final long DEFAULT_WRITE_TIMEOUT = 10_000;

    private String apiEndPoint = DEFAULT_API_END_POINT;
    private long connectTimeout = DEFAULT_CONNECT_TIMEOUT;
    private long readTimeout = DEFAULT_READ_TIMEOUT;
    private long writeTimeout = DEFAULT_WRITE_TIMEOUT;

    // okhttp 的攔截器
    private List interceptors = new ArrayList<>();

    private OkHttpClient.Builder okHttpClientBuilder;
    private Retrofit.Builder retrofitBuilder;

    // singleton 
    private SimSimiServiceBuilder(List interceptors) {
        this.interceptors.addAll(interceptors);
    }

    /**
     * Create a new {@link SimSimiServiceBuilder}
     */
    public static SimSimiServiceBuilder create() {
        return new SimSimiServiceBuilder(defaultInterceptors());
    }

    /**
     * 預設的攔截器, 放入 Log 攔截器, 
     * Log Request 跟 Resoponse 相關資訊
     */
    private static List defaultInterceptors() {
        final Logger slf4jLogger = LoggerFactory.getLogger("tw.com.dum");
        final HttpLoggingInterceptor httpLoggingInterceptor =
                new HttpLoggingInterceptor(message -> slf4jLogger.info("{}", message));
        httpLoggingInterceptor.setLevel(HttpLoggingInterceptor.Level.BODY);

        return Arrays.asList(
                new HeaderInterceptor(),
                httpLoggingInterceptor
        );
    }

    /**
     * RetrofitBuilder
     */
    private static Retrofit.Builder createDefaultRetrofitBuilder() {
        final ObjectMapper objectMapper = new ObjectMapper();
        objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);

        // Register JSR-310(java.time.temporal.*) module and read number as millsec.
        objectMapper.registerModule(new JavaTimeModule())
                .configure(DeserializationFeature.READ_DATE_TIMESTAMPS_AS_NANOSECONDS, false);

        return new Retrofit.Builder()
                .addConverterFactory(JacksonConverterFactory.create(objectMapper));
    }

    /**
     * 允許彈性的更改 api end point
     * Set apiEndPoint.
     */
    public SimSimiServiceBuilder apiEndPoint(@NonNull String apiEndPoint) {
        this.apiEndPoint = apiEndPoint;
        return this;
    }

    /**
     * 允許彈性的更改 connect timeout
     * Set connectTimeout in milliseconds.
     */
    public SimSimiServiceBuilder connectTimeout(long connectTimeout) {
        this.connectTimeout = connectTimeout;
        return this;
    }

    /**
     * 允許彈性的更改 read timeout
     * Set readTimeout in milliseconds.
     */
    public SimSimiServiceBuilder readTimeout(long readTimeout) {
        this.readTimeout = readTimeout;
        return this;
    }

    /**
     * 允許彈性的更改 write timeout
     * Set writeTimeout in milliseconds.
     */
    public SimSimiServiceBuilder writeTimeout(long writeTimeout) {
        this.writeTimeout = writeTimeout;
        return this;
    }

    /**
     * 允許彈性的增加 Interceptor
     * Add interceptor
     */
    public SimSimiServiceBuilder addInterceptor(Interceptor interceptor) {
        this.interceptors.add(interceptor);
        return this;
    }

    /**
     * 允許彈性的增加 First Interceptor
     * Add interceptor first
     */
    public SimSimiServiceBuilder addInterceptorFirst(Interceptor interceptor) {
        this.interceptors.add(0, interceptor);
        return this;
    }

    /**
     * If you want to use your own setting, specify {@link OkHttpClient.Builder} instance.
     */
    public SimSimiServiceBuilder okHttpClientBuilder(@NonNull OkHttpClient.Builder okHttpClientBuilder) {
        this.okHttpClientBuilder = okHttpClientBuilder;
        return this;
    }

    /**
     * If you want to use your own setting, specify {@link Retrofit.Builder} instance.
     * ref: {@link LineMessagingServiceBuilder#createDefaultRetrofitBuilder()} ()}.
     */
    public SimSimiServiceBuilder retrofitBuilder(@NonNull Retrofit.Builder retrofitBuilder) {
        this.retrofitBuilder = retrofitBuilder;
        return this;
    }

    /**
     * Creates a new {@link SimSimiService}.
     */
    public SimSimiService build() {
        // check and set HttpClient Builder
        if (okHttpClientBuilder == null) {
            okHttpClientBuilder = new OkHttpClient.Builder();
            interceptors.forEach(okHttpClientBuilder::addInterceptor);
        }
        okHttpClientBuilder
                .connectTimeout(connectTimeout, TimeUnit.MILLISECONDS)
                .readTimeout(readTimeout, TimeUnit.MILLISECONDS)
                .writeTimeout(writeTimeout, TimeUnit.MILLISECONDS);
        final OkHttpClient okHttpClient = okHttpClientBuilder.build();

        // check and set Retrofit Builder
        if (retrofitBuilder == null) {
            retrofitBuilder = createDefaultRetrofitBuilder();
        }
        retrofitBuilder.client(okHttpClient);
        retrofitBuilder.baseUrl(apiEndPoint);
        final Retrofit retrofit = retrofitBuilder.build();

        // Retrofit implements SimSimiService interface
        return retrofit.create(SimSimiService.class);
    }
}

/**
 * 這邊用 攔截器 替 Request 附加 Header
 * 附加 User-Agent 或是一些固定會用的 Header
 */
class HeaderInterceptor implements Interceptor {
    // user_agent, 在 http 通訊時告知自己的身份, 一堆爬蟲都會偽造自己是瀏覽器, 也都會改這個欄位 
    private static final String USER_AGENT =
            "my-botsdk-java/" + HeaderInterceptor.class.getPackage().getImplementationVersion();
    
    // channelToken 是 line-bot-sdk 必備的 header
    private final String channelToken;

    HeaderInterceptor(String channelToken) {
        this.channelToken = channelToken;
    }

    @Override
    public Response intercept(Chain chain) throws IOException {
        Request request = chain.request().newBuilder()
                               .addHeader("Authorization", "Bearer " + channelToken)
                               .addHeader("User-Agent", USER_AGENT)
                               .build();
        return chain.proceed(request);
    }
}

defaultInterceptors() 裡面有個 new HeaderInterceptor(),
當Request 透過發送出去的時候,你可以自定義自己的 Interceptor(攔截器),
去添加過濾或者做一些手腳, 比如說 token verify 的動作或 secret key 加密之類的,
透過 Interceptor 來處理, 就不用在商業邏輯處理這些鳥事。

Singleton Bean

早期的 Spring 的 Bean 都是要寫一堆臭臭的 xml,
在某個版本之後透過 @Configuration 的方式就可以取代臭臭的 xml,
是我比較喜歡的做法, Bean 的 Dependency Injection 特性,
可以讓我們要在商業邏輯中使用 SimSimiService 的時候,
只要給個 @Inject 或是 @Autowired 就可以實例化後注入。


public class MyBusinessService {

    private SimSimiService simService;

    @Inject
    public MyBusinessService(SimSimiService simService) {
        this.simService = simService;
    }
}

Configuration 裡面, 我放了 TRIAL_KEY 其實是可以從外部 application.yml 讀進來,
defaultQueryInterceptor 也應該從外面注入, 可以讓 Configuration 更單純一點。


/**
 * Created by jerry on 2017/1/9.
 */
@Configuration
public class Configuration {
    
    private static final String TRIAL_KEY = "foo-bar";
    
    /**
     * 沒有 zh-tw 嗚嗚嗚...
     * @see http://developer.simsimi.com/lclist
     */
    private static final String LOCATION_LAN = "ch";

    /**
     * 設定講髒話的程度, 忘了多少到多少
     */
    private static final String BAD_WORD_DISCRIMINATOR = "1.0";

    @Bean
    public SimSimiService simSimiService() {
        return SimSimiServiceBuilder
                .create()
                .addInterceptor(defaultQueryInterceptor())
                .build();
    }

    /**
     * 給定預設的 query, key, lc, ft
     *
     * @return
     */
    private Interceptor defaultQueryInterceptor() {
        return chain -> {
            HttpUrl keyQuery = chain.request().url()
                    .newBuilder()
                    .addQueryParameter("key", TRIAL_KEY)
                    .addQueryParameter("lc", LOCATION_LAN)
                    .addQueryParameter("ft", BAD_WORD_DISCRIMINATOR)
                    .build();
            Request request = chain.request().newBuilder().url(keyQuery).build();
            return chain.proceed(request);
        };
    }
}

測試

寫到這邊應該就可以寫個簡單的測試。
因為 Retrofit 用了 Callables 的方式來包裝每個 REST Request,
Callables 是是 Runnable 的進階版, 多允許在 multi-threads 下回傳 result,
這方便我們處理 Request 成功或失敗。


/**
 * Created by jerry on 2017/1/5.
 */
@RunWith(SpringRunner.class)
@SpringBootTest
public class SimSimServiceTest {

    /**
     * 注入的另一個好處是讓測試好方便
     */
    @Inject
    private SimSimiService simSimService;

    /**
     * Assert Response Code is 200
     */
    @Test
    public void simSimiApiTest() throws Exception {
        Response response = simSimService.chat("HI").execute();
        Assert.assertEquals(200, response.code());
    }
}

Batter than batter

雖然這樣在使用上已經很方便了, 但看到 line-bot-sdk 又多包了一層 servieClient 的東西,
多包這一層的用意是方便 DTO (Data Transfer Object) 的操作,
這邊細節又可以談很多, 下一篇吧。