Merge pull request #52 from github/more-documentation

more documentation updates

Author: Shlomi Noach

README.md

@@ -12,7 +12,22 @@
 
 WORK IN PROGRESS
 
-Please meanwhile refer to the [docs](doc) for more information.
+Please meanwhile refer to the [docs](doc) for more information. No, really, go to the [docs](doc). 
+
+## Usage
+
+#### Testing on replica
+
+```
+gh-ost --conf=.my.cnf --database=mydb --table=mytable --verbose --alter="engine=innodb" --execute --initially-drop-ghost-table --initially-drop-old-table -max-load=Threads_connected=30 --switch-to-rbr --chunk-size=2500 --cut-over=two-step --exact-rowcount --test-on-replica --verbose
+```
+Please read more on [testing on replica](testing-on-replica.md)
+
+#### Executing on master
+
+```
+gh-ost --conf=.my.cnf --database=mydb --table=mytable --verbose --alter="engine=innodb" --execute --initially-drop-ghost-table --initially-drop-old-table -max-load=Threads_connected=30 --switch-to-rbr --chunk-size=2500 --cut-over=two-step --exact-rowcount --verbose
+```
 
 ## What's in a name?
 

doc/command-line-flags.md

@@ -2,6 +2,10 @@
 
 A more in-depth discussion of various `gh-ost` command line flags: implementation, implication, use cases.
 
+##### cut-over
+
+Required. You are asked to explicitly state which cut-over algorithm you wish to use. Please see more discussion on [cut-over](cut-over.md)
+
 ##### exact-rowcount
 
 A `gh-ost` execution need to copy whatever rows you have in your existing table onto the ghost table. This can, and often be, a large number. Exactly what that number is?
@@ -14,3 +18,11 @@ A `gh-ost` execution need to copy whatever rows you have in your existing table
   We heuristically update the number of rows based on the queries we process from the binlogs.
 
 While the ongoing estimated number of rows is still heuristic, it's almost exact, such that the reported  [ETA](understanding-output.md) or percentage progress is typically accurate to the second throughout a multiple-hour operation.
+
+##### noop
+
+Do cheks; create ghost table and verify migration would be valid, but do not actually migrate and do not touch data.
+
+##### test-on-replica
+
+Issue the migration on a replica; do not modify data on master. Useful for validating, testing and benchmarking. See [test-on-replica](test-on-replica.md)

doc/cut-over.md

@@ -0,0 +1,14 @@
+# Cut-over step
+
+The cut-over is the final major step of the migration: it's the moment where your original table is pushed aside, and the ghost table (the one we secretly altered and operated on throughout the process) takes its place.
+
+MySQL poses some limitations on how the table swap can take place. While it supports an atomic swap, it does not allow for a swap under controlled lock.
+
+The [facebook OSC](https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932/) tool documents this nicely. Look for **"Cut-over phase"**.
+
+`gh-ost` supports various types of cut-over options:
+
+- `--cut-over=two-step` - this method is similar to the one taken by the facebook OSC. It's non-blocking but also non-atomic. The original table is first renames and pushed aside, then the ghost table is renamed to take its place. In between the two renames there's a brief period of time where your table just does not exist, and queries will fail.
+- `--cut-over=voluntary-lock` - as depicted in [Solving the Facebook-OSC non-atomic table swap problem](http://code.openark.org/blog/mysql/solving-the-facebook-osc-non-atomic-table-swap-problem), this solution uses voluntary MySQL locks, and makes for a blocking swap, where your queries do not fail, but block until operation is complete. This effect is desired. There is danger in this solution, since connection failure of the two sessions involved in creating the lock, would result in a premature swap of the tables, hence with potentially corrupted data.
+- We are working at this time on a blocking, safe, atomic solution, using wait conditions and via User Defined Functions which will need to be dynamically loaded onto your MySQL server. See [udf-ghost-wait-condition](https://github.com/openark/udf-ghost-wait-condition)
+- With [`--test-on-replica`](testing-on-replica.md) there is no table swap (**NOTE**: this is going to change. We will do cut-over on replica in exact same way as on master so as to invoke exact flow)

doc/swapping-tables.md

@@ -47,12 +47,12 @@ It's your job to:
 
 Simple:
 ```shell
-$ gh-osc --host=myhost.com --conf=/etc/gh-ost.cnf --database=test --table=sample_table --alter="engine=innodb" --chunk-size=2000 --max-load=Threads_connected=20 --initially-drop-ghost-table --initially-drop-old-table --test-on-replica --verbose --execute
+$ gh-osc --host=myhost.com --conf=/etc/gh-ost.cnf --database=test --table=sample_table --alter="engine=innodb" --chunk-size=2000 --max-load=Threads_connected=20 --initially-drop-ghost-table --initially-drop-old-table --cut-over=voluntary-lock --test-on-replica --verbose --execute
 ```
 
 Elaborate:
 ```shell
-$ gh-osc --host=myhost.com --conf=/etc/gh-ost.cnf --database=test --table=sample_table --alter="engine=innodb" --chunk-size=2000 --max-load=Threads_connected=20 --switch-to-rbr --initially-drop-ghost-table --initially-drop-old-table --test-on-replica --postpone-swap-tables-flag-file=/tmp/ghost-postpone.flag --exact-rowcount --allow-nullable-unique-key --verbose --execute
+$ gh-osc --host=myhost.com --conf=/etc/gh-ost.cnf --database=test --table=sample_table --alter="engine=innodb" --chunk-size=2000 --max-load=Threads_connected=20 --switch-to-rbr --initially-drop-ghost-table --initially-drop-old-table --cut-over=voluntary-lock --test-on-replica --postpone-swap-tables-flag-file=/tmp/ghost-postpone.flag --exact-rowcount --allow-nullable-unique-key --verbose --execute
 ```
 - Count exact number of rows (makes ETA estimation very good). This goes at the expense of paying the time for issuing a `SELECT COUNT(*)` on your table. We use this lovingly.
 - Automatically switch to `RBR` if replica is configured as `SBR`. See also: [migrating with SBR](migrating-with-sbr.md)

doc/testing-on-replica.md

@@ -134,5 +134,15 @@ Copy: 4466492/4467235 100.0%; Applied: 23309; Backlog: 1/100; Elapsed: 7m18s(cop
 2016-05-19 18:04:30 INFO Done waiting for events up to lock
 Copy: 4466492/4467235 100.0%; Applied: 23309; Backlog: 0/100; Elapsed: 7m18s(copy), 7m25s(total); streamer: mysql-bin.002589:17703056; ETA: 0s
 ```
-This migration took, till this point, `7m25s`, had applied `23309` events from the binary log and has copied `4466492` rows onto the ghost table.
- 
+This migration - insofar - took `7m25s`, has applied `23309` events from the binary log and has copied `4466492` rows onto the ghost table.
+
+## Status hint
+
+In addition, once every `10` minutes, a friendly reminder is printed, in the following form:
+
+```
+# Migrating `mydb`.`mytable`; Ghost table is `mydb`.`_mytable_gst`
+# Migration started at Mon Jun 06 03:45:08 -0700 2016
+# chunk-size: 2500; max lag: 1500ms; max-load: map[Threads_connected:30]
+# Throttle additional flag file: /tmp/gh-ost.throttle
+```