使用backup异地备份edX数据

Don’t put all your eggs in one basket

为了提升逼格，我决定使用英文版的的这句作为开头，尽管我最初想到的是中文版orz

天灾人祸这种事，总归是有的。尽管服务器的灾难性故障不常发生，概率也比遇到鬼或是遇到爱情来得高些，将风险考虑在内总归是好的，所谓有备无患。如果你已经在自己的架构中将单点故障考虑进去了，就不必往下拉啦，继续舔酸奶盖去吧~如果你的云服务商已经考虑了单点故障，你也没啥好做啦。

如果两者都没有，你应该请我喝杯星冰乐再往下拉。

ps：我最近多是在处理性能优化，数据迁移，安全策略，系统稳定性这类事，简直是伪装成开发的运维orz

#策略采用异地备份来应对单点故障导致的数据丢失

#最初的思路首先明确我们需要备份哪些数据,关于edX数据，可以参考我此前的这篇文章

一番深(hu)思(si)熟(luan)虑(xiang)之后，你会发现，我们关心的数据无非是数据库(mongodb/mysql)里存的，以及静态文件(/edx/var/edxapp/staticfiles)

思路很简单，每天定时备份数据库和静态文件，将其同步到远程服务器。只保存最近3，5天的备份文件，逾期的删除

这里需要注意的问题可能有三点：

不要影响到用户使用，备份是消耗服务器资源的，会造成服务器压力，写个定时任务，让其在凌晨做
采用增量同步的方式节省带宽
任务完成后，给自己发个邮件，确定成功执行了

那么写个shell脚本基本就完事啦，主要可能涉及的工具：

mongodump
mysqldump
tar
scp/rsync
crontab

#backup闪亮登场就在我shell脚本快写完的时候，在github上发现了这个东西：backup

一言话概括就是

Easy full stack backup operations on UNIX-like systems

稍微扫一眼Overview,分分钟爱不释手啊。

从文档中不难发现，backup本质上是对以上我们提到的工具做了封装。

能不写shell，我是尽量不写shell的。

先读下文档吧。

##install backup backup是ruby写的，使用gem安装。不得不说ruby的DSL能力真强啊

sudo gem install backup

哦，顺便说下，我在ruby1.8.7和ruby1.9.3下都没成功，之后在ubuntu12.04上手动编译了ruby2.0.0才安装好backup。也是蛮诡异的，可能是我机器的问题，文档里说ruby1.8.7和ruby1.9.3应该都是没有问题的

安装新版本的ruby建议用rvm，可以参考Ruby/Nodejs解释器版本依赖笔记

##backup demo 从一个简单的案例开始。

任务描述：备份我的home目录下的所有文件，同步到远程服务器的~/backup_my_home目录，只保留最近的5个备份

在shell里执行

1
2
3


:::text
backup generate:model --trigger backup_my_home \
  --archives --storages='scp'  --compressor='gzip'

会生成:Generated model file: '/home/USERNAME/Backup/models/backup_my_home.rb'

编辑以上文件就行

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# encoding: utf-8
Model.new(:backup_my_home, 'Description for backup_my_home') do

  archive :my_archive do |archive|
    # Run the `tar` command using `sudo`
    # archive.use_sudo
    archive.add "/home/USERNAME"
  end
  
  # SCP (Secure Copy) [Storage]
  #
  store_with SCP do |server|
    server.username   = "my_username"
    server.password   = "my_password"
    server.ip         = "123.45.678.90"
    server.port       = 22
    server.path       = "~/backup_my_home/"
    server.keep       = 5
  end

  # Gzip [Compressor]
  compress_with Gzip

end

几乎无需解释对吧:)

开始备份:backup perform -t backup_my_home

#开始备份edX

##静态文件

1
2
3


:::text
backup generate:model --trigger edx_staticfile_backup \
  --archives --storages='scp'  --compressor='gzip'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


# encoding: utf-8
Model.new(:edx_staticfile_backup, 'Description for edx_staticfile_backup') do


  #   archive.root '/path/to/archive/root'
  #
  archive :my_archive do |archive|
    # Run the `tar` command using `sudo`
    # archive.use_sudo
    #archive.add "/path/to/a/file.rb"
    archive.add "/edx/var/edxapp/staticfiles"
    #archive.exclude "/path/to/a/excluded_file.rb"
    #archive.exclude "/path/to/a/excluded_folder"
  end

  
  ##
  # SCP (Secure Copy) [Storage]
  #你需要先去目标服务器建立~/edx_backups/staticfile目录
  store_with SCP do |server|
    server.username   = "my_username"
    server.password   = "my_password"
    server.ip         = "123.45.678.90"
    server.port       = 22
    server.path       = "~/edx_backups/staticfile"
    server.keep       = 5

    # Additional options for the SSH connection.
    # server.ssh_options = {}
  end

  ##
  # Gzip [Compressor]
  #
  compress_with Gzip 

end

##备份mongo

##databases

1
2
3
4
5
6
7


:::text
>mongo
>show dbs
admin                            
cs_comments_service_development  
edxapp                           
local                            

##为mongo添加用户

1
2
3
4


:::text
>mongo
>use edxapp
>db.addUser(“wwj”, “xxx”)

1
2
3


:::text
backup generate:model --trigger edx_mongodb_backup \
  --databases="mongodb" --storages='scp'  --compressor='gzip' --notifiers='mail'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57


# encoding: utf-8
Model.new(:edx_mongodb_backup, 'Description for edx_mongodb_backup') do

  ##
  # MongoDB [Database]
  #
  database MongoDB do |db|
    db.name              = "edxapp"
    #db.name						= :all
    db.username           = "wwj"
    db.password           = "xxx"
    db.host               = "localhost"
    db.port               = 27017
    db.ipv6               = false
    #db.only_collections   = ["only", "these", "collections"]
    #db.additional_options = []
    db.lock               = false
    db.oplog              = false
  end

  ##
  # SCP (Secure Copy) [Storage]
  #
  store_with SCP do |server|
    server.username   = "my_username"
    server.password   = "my_password"
    server.ip         = "123.45.678.90"
    server.port       = 22
    server.path       = "~/edx_backups/mongo"
    server.keep       = 5

    # Additional options for the SSH connection.
    # server.ssh_options = {}
  end

  ##
  # Gzip [Compressor]
  #
  compress_with Gzip

  notify_by Mail do |mail|
    mail.on_success           = true
    mail.on_warning           = true
    mail.on_failure           = true

    mail.from                 = "sender@email.com"
    mail.to                   = "receiver@email.com"
    mail.address              = "smtp.gmail.com"
    mail.port                 = 587
    mail.domain               = "your.host.name"
    mail.user_name            = "sender@email.com"
    mail.password             = "my_password"
    mail.authentication       = "plain"
    mail.encryption           = :starttls
  end

end

##备份mysql

###databases

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


text
>mysql -u root
>show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| edxapp             |
| mysql              |
| ora                |
| performance_schema |
| test               |
| xqueue             |
+--------------------+

1
2
3


:::text
backup generate:model --trigger edx_mysql_backup \
  --databases="mysql" --storages='scp'  --compressor='gzip'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


# encoding: utf-8

Model.new(:edx_mysql_backup, 'Description for edx_mysql_backup') do

  ##
  # MySQL [Database]
  #
  database MySQL do |db|
    # To dump all databases, set `db.name = :all` (or leave blank)
    #db.name              = "my_database_name"
    db.name               = "edxapp"
    db.username           = "root"
    db.password           = ""
    db.host               = "localhost"
    db.port               = 3306
    #mysql.sock  位置有变
    #db.socket             = "/tmp/mysql.sock"
    # Note: when using `skip_tables` with the `db.name = :all` option,
    # table names should be prefixed with a database name.
    # e.g. ["db_name.table_to_skip", ...]
    #db.skip_tables        = ["skip", "these", "tables"]
    #db.only_tables        = ["only", "these", "tables"]
    db.additional_options = ["--quick", "--single-transaction"]
  end

  ##
  # SCP (Secure Copy) [Storage]
  #
  store_with SCP do |server|
    server.username   = "my_username"
    server.password   = "my_password"
    server.ip         = "123.45.678.90"
    server.port       = 22
    server.path       = "~/edx_backups/mysql"
    server.keep       = 5

    # Additional options for the SSH connection.
    # server.ssh_options = {}
  end

  ##
  # Gzip [Compressor]
  #
  compress_with Gzip 
end

##开始备份

1
2
3
4


:::text
backup perform -t edx_staticfile_backup
backup perform -t edx_mongodb_backup
backup perform -t edx_mysql_backup

##定时任务每天凌晨3:00

sudo crontab -e写入

1
2
3
4


:::text
00 03 * * * /usr/local/bin/backup perform -t edx_staticfile_backup
00 03 * * * /usr/local/bin/backup perform -t edx_mongodb_backup
00 03 * * * /usr/local/bin/backup perform -t edx_mysql_backup

#后续之后我们还是采用rsync的做法吧，不然数据大到一定程度，太浪费带宽了

只要更改store_with就行

注意：数据服务器到备份服务器应当配置好ssh免登陆

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


  compress_with Gzip do |gzip|
    gzip.rsyncable = true
  end

  store_with RSync do |storage|
    time = Time.now
    storage.mode = :ssh
    storage.host = "123.45.67.89"
    storage.port = 22
    storage.ssh_user = "wwj"
    #免密码ssh
    #每周为一轮，使用的是同名覆盖
    path = time.strftime '%A'
    storage.path = "~/edx_backups/staticfile/#{ path }"

  end

#注意 ~/Backup/log/backup.log 中的时间是ruby的时间，与本地差（晚）8个小时。时区的问题

文章目录