You've already forked minio-cookbook
mirror of
https://github.com/astaxie/cookbook.git
synced 2025-08-08 09:42:04 +03:00
Doc: Realigned the content home and moved cookbook content inside docs directory. (#45)
This commit is contained in:
koolhead17
committed by
Harshavardhana
Harshavardhana
parent
55a8a5bd21
commit
29546a45e8
114
docs/aggregate-apache-logs-with-fluentd-and-minio.md
Normal file
114
docs/aggregate-apache-logs-with-fluentd-and-minio.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# Store Apache Logs into Minio [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
In this recipe, we will learn how to use Minio as log aggregator for fluentd using `fluent-plugin-s3` plugin.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
* Install Minio Server from [here](http://docs.minio.io/docs/minio).
|
||||
* Install `mc` from [here](http://docs.minio.io/docs/minio-client-quick-start-guide)
|
||||
|
||||
## 2. Installation
|
||||
|
||||
* Install and run [Apache](https://httpd.apache.org) server.
|
||||
* Install [fluentd](http://docs.fluentd.org/articles/install-by-deb) and [fluent-plugin-s3](http://docs.fluentd.org/articles/apache-to-s3#amazon-s3-output).
|
||||
|
||||
|
||||
## 3. Recipe Steps
|
||||
|
||||
### Step 1: Create a bucket.
|
||||
|
||||
This is the bucket where fluentd will aggregate semi-structured apache logs in real-time.
|
||||
|
||||
```sh
|
||||
|
||||
$ mc mb myminio/fluentd
|
||||
Bucket created successfully ‘myminio/fluentd’.
|
||||
|
||||
```
|
||||
|
||||
### Step 2: Modify the fluentd configuration to use Minio as backend.
|
||||
Replace with your own values for `aws_key_id`, `aws_sec_key`, `s3_bucket`, `s3_endpoint`.
|
||||
|
||||
Replace `/etc/td-agent/td-agent.conf` with:
|
||||
|
||||
```sh
|
||||
|
||||
<source>
|
||||
@type tail
|
||||
format apache2
|
||||
path /var/log/apache2/access.log
|
||||
pos_file /var/log/td-agent/apache2.access.log.pos
|
||||
tag s3.apache.access
|
||||
</source>
|
||||
|
||||
<match>
|
||||
@type s3
|
||||
aws_key_id `aws_key_id`
|
||||
aws_sec_key `aws_sec_key`
|
||||
s3_bucket `s3_bucket`
|
||||
s3_endpoint `s3_endpoint`
|
||||
path logs/
|
||||
force_path_style true
|
||||
buffer_path /var/log/td-agent/s3
|
||||
time_slice_format %Y%m%d%H%M
|
||||
time_slice_wait 10m
|
||||
utc
|
||||
buffer_chunk_limit 256m
|
||||
</match>
|
||||
|
||||
```
|
||||
|
||||
### Step 3: Restart `fluentd` server.
|
||||
|
||||
```sh
|
||||
|
||||
sudo /etc/init.d/td-agent restart
|
||||
|
||||
|
||||
```
|
||||
|
||||
### Step 4: Check the fluentd logfile to confirm if everything is running.
|
||||
|
||||
|
||||
```sh
|
||||
|
||||
$ tail -f /var/log/td-agent/td-agent.log
|
||||
path logs/
|
||||
force_path_style true
|
||||
buffer_path /var/log/td-agent/s3
|
||||
time_slice_format %Y%m%d%H%M
|
||||
time_slice_wait 10m
|
||||
utc
|
||||
buffer_chunk_limit 256m
|
||||
</match>
|
||||
</ROOT>
|
||||
2016-05-03 18:44:44 +0530 [info]: following tail of /var/log/apache2/access.log
|
||||
|
||||
```
|
||||
|
||||
### Step 5: Test the configuration.
|
||||
|
||||
Ping the Apache server. This example uses the ab (Apache Bench) program.
|
||||
|
||||
|
||||
```sh
|
||||
|
||||
$ ab -n 100 -c 10 http://localhost/
|
||||
|
||||
```
|
||||
|
||||
### Step 6: Verify Aggregated Logs.
|
||||
|
||||
Minio server's fluent bucket should show the aggregated logs.
|
||||
|
||||
```sh
|
||||
|
||||
$ mc ls myminio/fluentd/logs/
|
||||
[2016-05-03 18:47:13 IST] 570B 201605031306_0.gz
|
||||
[2016-05-03 18:58:14 IST] 501B 201605031317_0.gz
|
||||
|
||||
```
|
||||
|
||||
**Note**:
|
||||
|
||||
fleuntd should have access permission for your apache log file located at `/var/log/apache2/access.log`.
|
||||
99
docs/aws-cli-with-minio.md
Normal file
99
docs/aws-cli-with-minio.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# AWS CLI with Minio Server [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
AWS CLI is a unified tool to manage AWS services. It is frequently the tool used to transfer data in and out of AWS S3. It works with any S3 compatible cloud storage service.
|
||||
|
||||
In this recipe we will learn how to configure and use AWS CLI to manage data with Minio Server.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Install Minio Server from [here](http://docs.minio.io/docs/minio).
|
||||
|
||||
## 2. Installation
|
||||
|
||||
Install AWS CLI from https://aws.amazon.com/cli/
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
To configure AWS CLI, type `aws configure` and specify the Minio key information.
|
||||
|
||||
Access credentials shown in this example belong to https://play.minio.io:9000.
|
||||
These credentials are open to public. Feel free to use this service for testing and development. Replace with your own Minio keys in deployment.
|
||||
|
||||
```sh
|
||||
|
||||
$ aws configure
|
||||
AWS Access Key ID [None]: Q3AM3UQ867SPQQA43P2F
|
||||
AWS Secret Access Key [None]: zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
|
||||
Default region name [None]: us-east-1
|
||||
Default output format [None]: ENTER
|
||||
|
||||
```
|
||||
|
||||
Additionally enable AWS Signature Version '4' for Minio server.
|
||||
|
||||
```sh
|
||||
|
||||
$ aws configure set default.s3.signature_version s3v4
|
||||
|
||||
```
|
||||
|
||||
## 4. Commands
|
||||
|
||||
### To list your buckets
|
||||
|
||||
```sh
|
||||
|
||||
$ aws --endpoint-url https://play.minio.io:9000 s3 ls
|
||||
2016-03-27 02:06:30 deebucket
|
||||
2016-03-28 21:53:49 guestbucket
|
||||
2016-03-29 13:34:34 mbtest
|
||||
2016-03-26 22:01:36 mybucket
|
||||
2016-03-26 15:37:02 testbucket
|
||||
|
||||
```
|
||||
|
||||
### To list contents inside bucket
|
||||
|
||||
```sh
|
||||
|
||||
$ aws --endpoint-url https://play.minio.io:9000 s3 ls s3://mybucket
|
||||
2016-03-30 00:26:53 69297 argparse-1.2.1.tar.gz
|
||||
2016-03-30 00:35:37 67250 simplejson-3.3.0.tar.gz
|
||||
|
||||
```
|
||||
|
||||
### To make a bucket
|
||||
|
||||
```sh
|
||||
|
||||
$ aws --endpoint-url https://play.minio.io:9000 s3 mb s3://mybucket
|
||||
make_bucket: s3://mybucket/
|
||||
|
||||
```
|
||||
|
||||
### To add an object to a bucket
|
||||
|
||||
```sh
|
||||
|
||||
$ aws --endpoint-url https://play.minio.io:9000 s3 cp simplejson-3.3.0.tar.gz s3://mybucket
|
||||
upload: ./simplejson-3.3.0.tar.gz to s3://mybucket/simplejson-3.3.0.tar.gz
|
||||
|
||||
```
|
||||
|
||||
### To delete an object from a bucket
|
||||
|
||||
```sh
|
||||
|
||||
$ aws --endpoint-url https://play.minio.io:9000 s3 rm s3://mybucket/argparse-1.2.1.tar.gz
|
||||
delete: s3://mybucket/argparse-1.2.1.tar.gz
|
||||
|
||||
```
|
||||
|
||||
### To remove a bucket
|
||||
|
||||
```sh
|
||||
|
||||
$ aws --endpoint-url https://play.minio.io:9000 s3 rb s3://mybucket
|
||||
remove_bucket: s3://mybucket/
|
||||
|
||||
```
|
||||
101
docs/aws-go-sdk-for-minio.md
Normal file
101
docs/aws-go-sdk-for-minio.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# AWS Go SDK for Minio Server [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
`aws-sdk-go` is the official AWS SDK for the Go programming language. In this recipe we will learn how to use `aws-sdk-go` with Minio server.
|
||||
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Install Minio Server from [here](http://docs.minio.io/docs/minio).
|
||||
|
||||
## 2. Installation
|
||||
|
||||
Install `aws-sdk-go` by:
|
||||
|
||||
```sh
|
||||
|
||||
|
||||
$ go get github.com/aws/aws-sdk-go/...
|
||||
|
||||
|
||||
```
|
||||
|
||||
## 3. Example
|
||||
|
||||
Access credentials shown in this example belong to https://play.minio.io:9000.
|
||||
These credentials are open to public. Feel free to use this service for testing and development. Replace with your own Minio keys in deployment.
|
||||
|
||||
List all buckets on minio server using aws-sdk-go.
|
||||
|
||||
```go
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/aws/credentials"
|
||||
"github.com/aws/aws-sdk-go/aws/session"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
)
|
||||
|
||||
func main() {
|
||||
newSession := session.New()
|
||||
|
||||
// Configure to use Minio Server
|
||||
s3Config := &aws.Config{
|
||||
Credentials: credentials.NewStaticCredentials("Q3AM3UQ867SPQQA43P2F", "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG", ""),
|
||||
Endpoint: aws.String("https://play.minio.io:9000"),
|
||||
Region: aws.String("us-east-1"),
|
||||
DisableSSL: aws.Bool(true),
|
||||
S3ForcePathStyle: aws.Bool(true),
|
||||
}
|
||||
|
||||
// Create an S3 service object in the default region.
|
||||
s3Client := s3.New(newSession, s3Config)
|
||||
|
||||
cparams := &s3.CreateBucketInput{
|
||||
Bucket: aws.String("newbucket"), // Required
|
||||
}
|
||||
|
||||
// Create a new bucket using the CreateBucket call.
|
||||
_, err := s3Client.CreateBucket(cparams)
|
||||
if err != nil {
|
||||
// Message from an error.
|
||||
fmt.Println(err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
var lparams *s3.ListBucketsInput
|
||||
|
||||
// Call the ListBuckets() Operation
|
||||
resp, err := s3Client.ListBuckets(lparams)
|
||||
if err != nil {
|
||||
// Message from an error.
|
||||
fmt.Println(err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Pretty-print the response data.
|
||||
fmt.Println(resp)
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
## 3. Run the Program
|
||||
|
||||
```sh
|
||||
|
||||
$ go run aws-sdk-minio.go
|
||||
{
|
||||
Buckets: [{
|
||||
CreationDate: 2015-10-22 01:46:04 +0000 UTC,
|
||||
Name: "newbucket"
|
||||
}],
|
||||
Owner: {
|
||||
DisplayName: "minio",
|
||||
ID: "minio"
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
@@ -0,0 +1,89 @@
|
||||
# Generate Let's Encrypt certificate using Concert for Minio [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
[Let’s Encrypt ](https://letsencrypt.org/) is a new free, automated, and open source, Certificate Authority.
|
||||
|
||||
[Concert ](https://docs.minio.io/docs/concert) is a console based certificate generation tool for Let’s Encrypt. It is open source & one of the related project from Minio.
|
||||
|
||||
In this recipe, we will generate a Let's Encypt certificate using Concert. This certificate will then be deployed for use in the Minio server.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
* Install Minio Server from [here](https://docs.minio.io/docs/minio).
|
||||
* Install Golang from [here](https://docs.minio.io/docs/how-to-install-golang).
|
||||
|
||||
## 2. Dependencies
|
||||
|
||||
* Port 443 for https needs to be open and available at time of executing `concert`.
|
||||
* Concert needs root access while executing because only root is allowed to bind to any port below 1024.
|
||||
* We will be using our own domain ``churchofminio.com`` as an example in this recipe. Replace with your own domain for your needs.
|
||||
|
||||
## 3. Recipe Steps
|
||||
|
||||
### Step 1: Install concert as shown below.
|
||||
|
||||
```sh
|
||||
|
||||
$ go get -u github.com/minio/concert
|
||||
|
||||
|
||||
```
|
||||
|
||||
### Step 2: Generate Let's Encrypt cert.
|
||||
|
||||
|
||||
```sh
|
||||
|
||||
$ sudo concert gen --dir my-certs admin@churchofminio.com churchofminio.com
|
||||
2016/04/04 07:10:01 Generated certificates for churchofminio.com under my-certs will expire in 89 days.
|
||||
|
||||
```
|
||||
|
||||
### Step 3: Verify Certificates.
|
||||
|
||||
List certs saved in `my-certs` directory.
|
||||
|
||||
```sh
|
||||
|
||||
$ ls -l my-certs/
|
||||
total 12
|
||||
-rw------- 1 root root 227 Apr 4 07:10 certs.json
|
||||
-rw------- 1 root root 1679 Apr 4 07:10 private.key
|
||||
-rw------- 1 root root 3448 Apr 4 07:10 public.crt
|
||||
|
||||
```
|
||||
|
||||
### Step 4: Set up SSL on Minio Server with the certificates.
|
||||
|
||||
The generated keys via Concert needs to be placed inside users home directory.
|
||||
|
||||
``${HOME}/.minio/certs``
|
||||
|
||||
```sh
|
||||
|
||||
$ cp my-certs/private.key /home/supernova/.minio/certs/
|
||||
$ cp my-certs/public.crt /home/supernova/.minio/certs/
|
||||
|
||||
```
|
||||
|
||||
### Step 5: Change ownership of certificates.
|
||||
|
||||
```sh
|
||||
|
||||
$ sudo chown supernova:supernova /home/supernova/.minio/certs/private.key
|
||||
$ sudo chown supernova:supernova /home/supernova/.minio/certs/public.crt
|
||||
|
||||
```
|
||||
|
||||
### Step 6: Start Minio Server using HTTPS.
|
||||
|
||||
Start Minio Server as shown below.
|
||||
|
||||
```sh
|
||||
|
||||
$ ./minio server export/
|
||||
|
||||
```
|
||||
|
||||
### Step 7: Visit https://churchofminio.com:9000 in the browser.
|
||||
|
||||
|
||||
121
docs/how-to-install-golang.md
Normal file
121
docs/how-to-install-golang.md
Normal file
@@ -0,0 +1,121 @@
|
||||
# How to install Golang? [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
## Ubuntu (Kylin) 14.04
|
||||
|
||||
### Build Dependencies
|
||||
|
||||
This installation document assumes Ubuntu 14.04+ on x86-64 platform.
|
||||
|
||||
##### Install Git, GCC
|
||||
|
||||
```sh
|
||||
|
||||
$ sudo apt-get install git build-essential
|
||||
|
||||
```
|
||||
|
||||
##### Install Go 1.6+
|
||||
|
||||
Download Go 1.6+ from [https://golang.org/dl/](https://golang.org/dl/).
|
||||
|
||||
```sh
|
||||
|
||||
$ wget https://storage.googleapis.com/golang/go1.6.linux-amd64.tar.gz
|
||||
$ mkdir -p ${HOME}/bin/
|
||||
$ mkdir -p ${HOME}/go/
|
||||
$ tar -C ${HOME}/bin/ -xzf go1.6.linux-amd64.tar.gz
|
||||
|
||||
```
|
||||
|
||||
##### Setup GOROOT and GOPATH
|
||||
|
||||
Add the following exports to your ``~/.bashrc``. Environment variable GOROOT specifies the location of your golang binaries
|
||||
and GOPATH specifies the location of your project workspace.
|
||||
|
||||
```sh
|
||||
|
||||
export GOROOT=${HOME}/bin/go
|
||||
export GOPATH=${HOME}/go
|
||||
export PATH=${HOME}/bin/go/bin:${GOPATH}/bin:$PATH
|
||||
|
||||
```
|
||||
|
||||
##### Source the new enviornment
|
||||
|
||||
```sh
|
||||
|
||||
$ source ~/.bashrc
|
||||
|
||||
```
|
||||
|
||||
##### Testing it all
|
||||
|
||||
```sh
|
||||
|
||||
$ go env
|
||||
|
||||
```
|
||||
|
||||
## OS X (Yosemite) 10.10
|
||||
|
||||
### Build Dependencies
|
||||
|
||||
This installation document assumes OS X Yosemite 10.10+ on x86-64 platform.
|
||||
|
||||
##### Install brew
|
||||
|
||||
```sh
|
||||
|
||||
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
|
||||
|
||||
```
|
||||
|
||||
##### Install Git, Python
|
||||
|
||||
```sh
|
||||
|
||||
$ brew install git python
|
||||
|
||||
```
|
||||
|
||||
##### Install Go 1.6+
|
||||
|
||||
Install golang binaries using `brew`
|
||||
|
||||
```sh
|
||||
|
||||
$ brew install go
|
||||
$ mkdir -p $HOME/go
|
||||
|
||||
```
|
||||
|
||||
##### Setup GOROOT and GOPATH
|
||||
|
||||
Add the following exports to your ``~/.bash_profile``. Environment variable GOROOT specifies the location of your golang binaries
|
||||
and GOPATH specifies the location of your project workspace.
|
||||
|
||||
```sh
|
||||
|
||||
export GOPATH=${HOME}/go
|
||||
export GOVERSION=$(brew list go | head -n 1 | cut -d '/' -f 6)
|
||||
export GOROOT=$(brew --prefix)/Cellar/go/${GOVERSION}/libexec
|
||||
export PATH=${GOPATH}/bin:$PATH
|
||||
|
||||
```
|
||||
|
||||
##### Source the new enviornment
|
||||
|
||||
```sh
|
||||
|
||||
$ source ~/.bash_profile
|
||||
|
||||
```
|
||||
|
||||
##### Testing it all
|
||||
|
||||
```sh
|
||||
|
||||
$ go env
|
||||
|
||||
```
|
||||
|
||||
106
docs/rclone-with-minio.md
Normal file
106
docs/rclone-with-minio.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# rclone with Minio Server [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
`rclone` is an open source command line program to sync files and
|
||||
directories to and from cloud storage systems. It aims to be "rsync
|
||||
for cloud storage".
|
||||
|
||||
This recipe describes how to use rclone with Minio Server.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
First install Minio Server from [minio.io](https://minio.io/).
|
||||
|
||||
## 2. Installation
|
||||
|
||||
Next install rclone from [rclone.org](http://rclone.org).
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
When it configures itself Minio will print something like this
|
||||
|
||||
```sh
|
||||
|
||||
AccessKey: WLGDGYAQYIGI833EV05A SecretKey: BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF Region: us-east-1
|
||||
|
||||
Minio Object Storage:
|
||||
http://127.0.0.1:9000
|
||||
http://10.0.0.3:9000
|
||||
|
||||
Minio Browser:
|
||||
http://127.0.0.1:9000
|
||||
http://10.0.0.3:9000
|
||||
|
||||
```
|
||||
|
||||
You now need to configure those details into rclone.
|
||||
|
||||
Run `rclone config`, create a new remote called `minio` (or anything
|
||||
else) of type `S3` and enter the details above something like this:
|
||||
|
||||
(Note that it is important to put the region in as stated above.)
|
||||
|
||||
```sh
|
||||
|
||||
env_auth> 1
|
||||
access_key_id> WLGDGYAQYIGI833EV05A
|
||||
secret_access_key> BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF
|
||||
region> us-east-1
|
||||
endpoint> http://10.0.0.3:9000
|
||||
location_constraint>
|
||||
server_side_encryption>
|
||||
|
||||
```
|
||||
|
||||
Which makes the config file look like this
|
||||
|
||||
```sh
|
||||
|
||||
[minio]
|
||||
env_auth = false
|
||||
access_key_id = WLGDGYAQYIGI833EV05A
|
||||
secret_access_key = BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF
|
||||
region = us-east-1
|
||||
endpoint = http://10.0.0.3:9000
|
||||
location_constraint =
|
||||
server_side_encryption =
|
||||
|
||||
```
|
||||
|
||||
## 4. Commands
|
||||
|
||||
Minio doesn't support all the features of S3 yet. In particular it
|
||||
doesn't support MD5 checksums (ETags) or metadata. This means rclone
|
||||
can't check MD5SUMs or store the modified date. However you can work
|
||||
around this with the `--size-only` flag of rclone.
|
||||
|
||||
Here are some example commands
|
||||
|
||||
List buckets
|
||||
|
||||
rclone lsd minio:
|
||||
|
||||
Make a new bucket
|
||||
|
||||
rclone mkdir minio:bucket
|
||||
|
||||
Copy files into that bucket
|
||||
|
||||
rclone --size-only copy /path/to/files minio:bucket
|
||||
|
||||
Copy files back from that bucket
|
||||
|
||||
rclone --size-only copy minio:bucket /tmp/bucket-copy
|
||||
|
||||
List all the files in the bucket
|
||||
|
||||
rclone ls minio:bucket
|
||||
|
||||
Sync files into that bucket - try with `--dry-run` first
|
||||
|
||||
rclone --size-only --dry-run sync /path/to/files minio:bucket
|
||||
|
||||
Then sync for real
|
||||
|
||||
rclone --size-only sync /path/to/files minio:bucket
|
||||
|
||||
See the [rclone web site](http://rclone.org) for more examples and docs.
|
||||
48
docs/restic-with-minio.md
Normal file
48
docs/restic-with-minio.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# restic with Minio Server [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
`restic` is a fast, efficient and secure backup tool. It is an open source project available under ``BSD 2-Clause License``.
|
||||
|
||||
In this recipe we will learn how to use `restic` to backup data into Minio Server.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Install Minio Server from [here](http://docs.minio.io/docs/minio).
|
||||
|
||||
## 2. Installation
|
||||
|
||||
Install restic from [https://restic.github.io](https://restic.github.io).
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
Set Minio credentials in the environment variables as shown below.
|
||||
|
||||
```sh
|
||||
|
||||
$ export AWS_ACCESS_KEY_ID=<YOUR-ACCESS-KEY-ID>
|
||||
$ export AWS_SECRET_ACCESS_KEY= <YOUR-SECRET-ACCESS-KEY>
|
||||
|
||||
```
|
||||
|
||||
## 4. Commands
|
||||
|
||||
Start `restic` and point it to the bucket where the backup data will reside.
|
||||
|
||||
```sh
|
||||
|
||||
$ ./restic -r s3:http://localhost:9000/resticbucket init
|
||||
|
||||
```
|
||||
|
||||
Copy backups from the local machine to the bucket on minio server.
|
||||
|
||||
```sh
|
||||
|
||||
$ ./restic -r s3:http://localhost:9000/resticbucket backup /home/minio/workdir/Docs/
|
||||
enter password for repository:
|
||||
scan [/home/minio/workdir/Docs]
|
||||
scanned 2 directories, 6 files in 0:00
|
||||
[0:00] 100.00% 0B/s 8.045 KiB / 8.045 KiB 6 / 8 items 0 errors ETA 0:00
|
||||
duration: 0:00, 0.06MiB/s
|
||||
snapshot 85a9731a saved
|
||||
|
||||
```
|
||||
16
docs/running-deis-workflow-with-minio.md
Normal file
16
docs/running-deis-workflow-with-minio.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# Running Deis Workflow with Minio [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
[Deis Workflow](https://deis.com/) is an open source [PaaS](https://en.wikipedia.org/wiki/Platform_as_a_service) that makes it easy to deploy and manage applications on your own servers. Workflow builds upon [Kubernetes](http://kubernetes.io/) and [Docker](https://www.docker.com/) to provide a lightweight PaaS with a [Heroku](https://www.heroku.com/)-inspired workflow. Workflow is implemented as a variety of self-contained components (see https://github.com/deis for a list) which communicate using both the Kubernetes system and an object storage server. It's configurable to use cloud object storage systems like [Amazon S3](https://aws.amazon.com/s3/), [Google Cloud Storage](https://cloud.google.com/storage/) and [Microsoft Azure Storage](https://azure.microsoft.com/en-us/services/storage/) and, of course, Minio. We don't yet recommend you use Minio in production Deis Workflow installations, we do recommend it as a great way to quickly install a Deis Workflow cluster for a quick demo, development, testing, etc. In fact, we ship Deis Workflow with Minio installed by default.
|
||||
|
||||
To use it, follow the instructions at https://docs-v2.readthedocs.io/en/latest/installing-workflow/. Once you've completed the installation, follow any of the three methods for deployment listed below:
|
||||
|
||||
- [Buildpack Deployment](https://docs-v2.readthedocs.io/en/latest/applications/using-buildpacks/)
|
||||
- [Dockerfile Deployment](https://docs-v2.readthedocs.io/en/latest/applications/using-dockerfiles/)
|
||||
- [Docker Image Deployment](https://docs-v2.readthedocs.io/en/latest/applications/using-docker-images/)
|
||||
|
||||
All of the three deployment methods, as well as Workflow internals use Minio extensively behind the scenes:
|
||||
|
||||
- Buildpack deployments use Minio to store code and [slugs](https://devcenter.heroku.com/articles/slug-compiler)
|
||||
- Dockerfile deployments use Minio to store Dockerfiles and associated artifacts
|
||||
- Docker Image deployments use Minio as the backing store for the internal Docker registry that Workflow runs
|
||||
- The internal Workflow database stores user login information, SSH keys, and more. It backs all data up to Minio
|
||||
107
docs/running-minio-in-docker.md
Normal file
107
docs/running-minio-in-docker.md
Normal file
@@ -0,0 +1,107 @@
|
||||
# How to run Minio in Docker? [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* You have Docker installed and running, if not follow [install instructions](https://docs.docker.com/engine/installation/ubuntulinux/)
|
||||
* You have minio client aka mc installed, if not follow [install instructions](https://docs.minio.io/docs/minio-client-quick-start-guide)
|
||||
|
||||
## Steps
|
||||
|
||||
### Add a local alias for docker (optional)
|
||||
|
||||
I am adding an [``alias``](http://tldp.org/LDP/abs/html/aliases.html) to my local ``bashrc`` file to avoid typing ``sudo`` along with running docker command.
|
||||
|
||||
```sh
|
||||
|
||||
alias docker="sudo /usr/bin/docker"
|
||||
|
||||
```
|
||||
|
||||
### Fetching Minio image from repository & running Minio in docker.
|
||||
|
||||
```sh
|
||||
|
||||
$ docker run -p 9000:9000 minio/minio:latest
|
||||
|
||||
AccessKey: IQP18YBF51DG8HSZEE7B SecretKey: AlDzw6dj9zfne8JhPwGapt0Idlfg/QLhMq58Z0ax
|
||||
|
||||
Starting minio server:
|
||||
Listening on http://127.0.0.1:9000
|
||||
Listening on http://172.17.0.3:9000
|
||||
|
||||
```
|
||||
|
||||
### Add Minio configuration to mc
|
||||
|
||||
```sh
|
||||
|
||||
$ mc config host add localhost http://localhost:9000 IQP18YBF51DG8HSZEE7B AlDzw6dj9zfne8JhPwGapt0Idlfg/QLhMq58Z0ax
|
||||
|
||||
```
|
||||
|
||||
### Play with Minio server
|
||||
|
||||
```sh
|
||||
|
||||
$ mc mb localhost/newbucket
|
||||
Bucket created successfully ‘localhost/newbucket’.
|
||||
$ mc mb localhost/mybucket
|
||||
Bucket created successfully ‘localhost/mybucket’.
|
||||
|
||||
```
|
||||
|
||||
### Persist Minio configs
|
||||
|
||||
Running Minio container
|
||||
|
||||
```sh
|
||||
|
||||
$ docker ps
|
||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||||
51e3a48d209a minio/minio:latest "/minio server /expo 22 hours ago Up 22 hours 0.0.0.0:9000->9000/tcp fervent_shockley
|
||||
|
||||
```
|
||||
|
||||
```sh
|
||||
|
||||
$ docker commit 51e3a48d209a minio/my-minio
|
||||
fcc98afd0b4da9340b3e635d73a82088e7224798b3467138840b997959af4520
|
||||
|
||||
$ docker stop 51e3a48d209a
|
||||
51e3a48d209a
|
||||
|
||||
```
|
||||
|
||||
> NOTE: Replace the container id with
|
||||
>your own.
|
||||
|
||||
#### Create a data volume container
|
||||
|
||||
```sh
|
||||
|
||||
$ docker create -v /export --name minio-export minio/my-minio /bin/true
|
||||
4e466c4572b96cc16a619f6e13155657745aa653b1857929100f1a8208a58da8
|
||||
$ docker run -p 9000:9000 --volumes-from minio-export --name minio1 minio/my-minio
|
||||
|
||||
AccessKey: IQP18YBF51DG8HSZEE7B SecretKey: AlDzw6dj9zfne8JhPwGapt0Idlfg/QLhMq58Z0ax
|
||||
|
||||
|
||||
Starting minio server:
|
||||
Listening on http://127.0.0.1:9000
|
||||
Listening on http://172.17.0.4:9000
|
||||
|
||||
```
|
||||
|
||||
#### Test the Persist feature
|
||||
|
||||
We created a few buckets in our previous Minio server. Let us see if they still exists.
|
||||
|
||||
```sh
|
||||
|
||||
$ mc ls localhost
|
||||
[2016-01-20 14:25:56 IST] 0B mybucket/
|
||||
[2016-01-21 12:59:52 IST] 0B newbucket/
|
||||
|
||||
```
|
||||
|
||||
They are very much intact, it clearly means we were able to store the Minio docker image.
|
||||
126
docs/s3cmd-with-minio.md
Normal file
126
docs/s3cmd-with-minio.md
Normal file
@@ -0,0 +1,126 @@
|
||||
# s3cmd with Minio Server [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
`s3cmd` is a CLI client for managing data in AWS S3, Google Cloud Storage or any cloud storage service provider that uses the s3 protocol. `s3cmd` is open source and is distributed under the GPLv2 license.
|
||||
|
||||
In this recipe we will learn how to configure and use s3cmd to manage data with Minio Server.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Install Minio Server from [here](http://docs.minio.io/docs/minio).
|
||||
|
||||
## 2. Installation
|
||||
|
||||
Install `s3cmd` from http://s3tools.org/s3cmd.
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
We will run `s3cmd` on https://play.minio.io:9000.
|
||||
|
||||
Access credentials shown in this example belong to https://play.minio.io:9000.
|
||||
These credentials are open to public. Feel free to use this service for testing and development. Replace with your own Minio keys in deployment.
|
||||
|
||||
Edit the following fields in your s3cmd configuration file `~/.s3cfg`
|
||||
|
||||
```sh
|
||||
|
||||
# Setup endpoint
|
||||
host_base = play.minio.io:9000
|
||||
host_bucket = play.minio.io:9000
|
||||
bucket_location = us-east-1
|
||||
use_https = True
|
||||
|
||||
# Setup access keys
|
||||
access_key = Q3AM3UQ867SPQQA43P2F
|
||||
secret_key = zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
|
||||
|
||||
# Enable S3 v4 signature APIs
|
||||
signature_v2 = False
|
||||
|
||||
```
|
||||
|
||||
## 4. Commands
|
||||
|
||||
### To make a bucket
|
||||
|
||||
```sh
|
||||
|
||||
$ s3cmd mb s3://mybucket
|
||||
Bucket 's3://mybucket/' created
|
||||
|
||||
```
|
||||
|
||||
### To copy an object to bucket
|
||||
|
||||
```sh
|
||||
|
||||
$ s3cmd put newfile s3://testbucket
|
||||
upload: 'newfile' -> 's3://testbucket/newfile'
|
||||
|
||||
```
|
||||
|
||||
### To copy an object to local system
|
||||
|
||||
```sh
|
||||
|
||||
$ s3cmd get s3://testbucket/newfile
|
||||
download: 's3://testbucket/newfile' -> './newfile'
|
||||
|
||||
```
|
||||
|
||||
### To sync local file/directory to a bucket
|
||||
|
||||
```sh
|
||||
|
||||
$ s3cmd sync newdemo s3://testbucket
|
||||
upload: 'newdemo/newdemofile.txt' -> 's3://testbucket/newdemo/newdemofile.txt'
|
||||
|
||||
```
|
||||
|
||||
### To sync bucket or object with local filesystem
|
||||
|
||||
```sh
|
||||
|
||||
$ s3cmd sync s3://testbucket otherlocalbucket
|
||||
download: 's3://testbucket/cat.jpg' -> 'otherlocalbucket/cat.jpg'
|
||||
|
||||
```
|
||||
|
||||
### To list buckets
|
||||
|
||||
```sh
|
||||
|
||||
$ s3cmd ls s3://
|
||||
2015-12-09 16:12 s3://testbbucket
|
||||
|
||||
```
|
||||
|
||||
### To list contents inside bucket
|
||||
|
||||
```sh
|
||||
|
||||
$ s3cmd ls s3://testbucket/
|
||||
DIR s3://testbucket/test/
|
||||
2015-12-09 16:05 138504 s3://testbucket/newfile
|
||||
|
||||
```
|
||||
|
||||
### To delete an object from bucket
|
||||
|
||||
```sh
|
||||
|
||||
$ s3cmd del s3://testbucket/newfile
|
||||
delete: 's3://testbucket/newfile'
|
||||
|
||||
```
|
||||
|
||||
### To delete a bucket
|
||||
|
||||
```sh
|
||||
|
||||
$ s3cmd rb s3://mybucket
|
||||
Bucket 's3://mybucket/' removed
|
||||
|
||||
```
|
||||
|
||||
NOTE:
|
||||
The complete usage guide for `s3cmd` is available [here](http://s3tools.org/usage).
|
||||
49
docs/setup-caddy-proxy-with-minio.md
Normal file
49
docs/setup-caddy-proxy-with-minio.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# Setup Caddy proxy with Minio Server [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
Caddy is a web server like Apache, nginx, or lighttpd. The purpose of Caddy is to streamline web development, deployment, and hosting workflows so that anyone can host their own web sites without requiring special technical knowledge.
|
||||
|
||||
In this recipe we will learn how to set up Caddy proxy with Minio Server.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Install Minio Server from [here](http://docs.minio.io/docs/minio).
|
||||
|
||||
## 2. Installation
|
||||
|
||||
Install Caddy Server from [here](https://caddyserver.com/download).
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
Create a caddy configuration file as below, change the ip addresses according to your local minio and DNS configuration.
|
||||
|
||||
```sh
|
||||
|
||||
your.public.com
|
||||
|
||||
proxy / localhost:9000 {
|
||||
transparent
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
## 4. Recipe Steps
|
||||
|
||||
### Step 1: Start `minio` server.
|
||||
|
||||
|
||||
```sh
|
||||
|
||||
$ ./minio --address localhost:9000 server <your_export_dir>
|
||||
|
||||
```
|
||||
|
||||
### Step 2: Start `caddy` server.
|
||||
|
||||
```sh
|
||||
|
||||
$ ./caddy
|
||||
Activating privacy features... done.
|
||||
your.public.com:443
|
||||
your.public.com:80
|
||||
|
||||
```
|
||||
55
docs/setup-nginx-proxy-with-minio.md
Normal file
55
docs/setup-nginx-proxy-with-minio.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# Setup Nginx proxy with Minio Server [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
Nginx is an open source Web server and a reverse proxy server.
|
||||
|
||||
In this recipe we will learn how to set up Nginx proxy with Minio Server.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Install Minio Server from [here](http://docs.minio.io/docs/minio).
|
||||
|
||||
## 2. Installation
|
||||
|
||||
Install Nginx from [here](http://nginx.org/en/download.html).
|
||||
|
||||
## 3. Configuration
|
||||
|
||||
Add below content as a file ``/etc/nginx/sites-enabled`` and also remove the existing ``default`` file in same directory.
|
||||
|
||||
```sh
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name example.com;
|
||||
location / {
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_pass http://localhost:9000;
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
Note:
|
||||
|
||||
* Replace example.com with your own hostname.
|
||||
* Replace ``http://localhost:9000`` with your own server name.
|
||||
|
||||
## 4. Recipe Steps
|
||||
|
||||
### Step 1: Start Minio server.
|
||||
|
||||
```sh
|
||||
|
||||
$ minio server /mydatadir
|
||||
|
||||
```
|
||||
|
||||
### Step 2: Restart Nginx server.
|
||||
|
||||
```sh
|
||||
|
||||
$ sudo service nginx restart
|
||||
|
||||
```
|
||||
51
docs/store-mongodb-backups-in-minio.md
Normal file
51
docs/store-mongodb-backups-in-minio.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# Store MongoDB Backups in Minio Server [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
In this recipe we will learn how to store MongoDB backups in Minio Server.
|
||||
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
* Install mc from [here](https://docs.minio.io/docs/minio-client-quick-start-guide).
|
||||
* Install Minio Server from [here](https://docs.minio.io/docs/minio ).
|
||||
* Know where the MongoDB backups reside in the local filesystem.
|
||||
|
||||
|
||||
## 2. Recipe Steps
|
||||
|
||||
In this recipe, we will use https://play.minio.io:9000 which is aliased to play. Feel free to use play server for testing and development. Access credentials shown in this example are open to public.
|
||||
Replace with your own access credentials when running this example in your environment.
|
||||
|
||||
### Step 1: Create a bucket.
|
||||
|
||||
```sh
|
||||
|
||||
$ mc mb play/mongobkp
|
||||
Bucket created successfully ‘play/mongobkp’.
|
||||
|
||||
```
|
||||
|
||||
### Step 2: Mirror local backup to Minio server.
|
||||
|
||||
```sh
|
||||
|
||||
$ mc mirror mongobkp/ play/mongobkp
|
||||
|
||||
|
||||
```
|
||||
|
||||
## 3. Automate
|
||||
|
||||
The above recipe can be automated easily. Change the bash script below to your own directories and PATHS as needed. Set up a cron to run this task as needed.
|
||||
|
||||
```sh
|
||||
|
||||
#!/bin/bash
|
||||
#FileName: Minimongobkp.sh & has executable permissions.
|
||||
|
||||
LocalBackupPath="/home/miniouser/mongobkp"
|
||||
MinioBucket="play/mongobkp"
|
||||
MCPATH="/home/miniouser/go/bin/mc"
|
||||
|
||||
$MCPATH - -quiet mirror $LocalBackupPath $MinioBucket
|
||||
|
||||
```
|
||||
49
docs/store-mysql-backups-in-minio.md
Normal file
49
docs/store-mysql-backups-in-minio.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# Store MySQL Backups in Minio Server [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
In this recipe we will learn how to store MySQL backups in Minio Server.
|
||||
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
* Install mc from [here](https://docs.minio.io/docs/minio-client-quick-start-guide).
|
||||
* Install Minio Server from [here](https://docs.minio.io/docs/minio ).
|
||||
* Know where the MySQL backups reside in the local filesystem.
|
||||
|
||||
## 2. Recipe Steps
|
||||
|
||||
Access credentials shown in this example belong to https://play.minio.io:9000.
|
||||
These credentials are open to public. Feel free to use this service for testing and development. Replace with your own Minio keys in deployment.
|
||||
|
||||
### Step 1: Create a bucket.
|
||||
|
||||
```sh
|
||||
|
||||
$ mc mb play/mysqlbkp
|
||||
Bucket created successfully ‘play/mysqlbkp’.
|
||||
|
||||
```
|
||||
|
||||
### Step 2: Mirror mysqlbkup directory where the backup files reside to Minio server.
|
||||
|
||||
```sh
|
||||
|
||||
$ mc mirror mysqlbkp/ play/mysqlbkp
|
||||
|
||||
```
|
||||
|
||||
## 3. Automate
|
||||
|
||||
The above recipe can be automated easily. Change the bash script below to your own directories and PATHS as needed. Set up a cron to run this task as needed.
|
||||
|
||||
```sh
|
||||
|
||||
#!/bin/bash
|
||||
# Filename: minio-mysql-bkp.sh & has executable permissions.
|
||||
|
||||
LOCAL_BACKUP_PATH="/home/miniouser/mysqlbkp"
|
||||
MINIO_BUCKET="play/mysqlbkp"
|
||||
MC_PATH="/home/miniouser/go/bin/mc"
|
||||
|
||||
$MC_PATH - -quiet mirror $LOCAL_BACKUP_PATH $MINIO_BUCKET
|
||||
|
||||
```
|
||||
48
docs/store-postgresql-backups-in-minio.md
Normal file
48
docs/store-postgresql-backups-in-minio.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# Store PostgreSQL Backups in Minio Server [](https://gitter.im/minio/minio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
||||
|
||||
In this recipe you will learn how to store PostgreSQL backups in Minio Server.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
* Install mc from [here](https://docs.minio.io/docs/minio-client-quick-start-guide).
|
||||
* Install Minio Server from [here](https://docs.minio.io/docs/minio ).
|
||||
* Know where the PostgreSQL backups reside in the local filesystem.
|
||||
|
||||
## 2. Recipe Steps
|
||||
|
||||
In this recipe, we will use https://play.minio.io:9000 which is aliased to play. Feel free to use play server for testing and development. Access credentials shown in this example are open to public.
|
||||
Replace with your own access credentials when running this example in your environment.
|
||||
|
||||
### Step 1: Create a bucket:
|
||||
|
||||
```sh
|
||||
|
||||
$ mc mb play/pgsqlbkp
|
||||
Bucket created successfully ‘play/pgsqlbkp’.
|
||||
|
||||
```
|
||||
|
||||
### Step 2: Mirror local backup to Minio server:
|
||||
|
||||
```sh
|
||||
|
||||
$ mc mirror pgsqlbkp/ play/pgsqlbkp
|
||||
|
||||
```
|
||||
|
||||
## 3. Automate
|
||||
|
||||
The above recipe can be automated easily. Change the bash script below to your own directories and PATHS as needed. Set up a cron to run this task as needed.
|
||||
|
||||
```sh
|
||||
|
||||
#!/bin/bash
|
||||
#FileName: Miniopgsqlbkp.sh & has executable permissions.
|
||||
|
||||
LocalBackupPath="/home/miniouser/pgsqlbkp"
|
||||
MinioBucket="play/pgsqlbkp"
|
||||
MCPATH="/home/miniouser/go/bin/mc"
|
||||
|
||||
$MCPATH - -quiet mirror $LocalBackupPath $MinioBucket
|
||||
|
||||
```
|
||||
Reference in New Issue
Block a user