What is a good logging practice for distributed tasks?

"Serverless" mostly just means you've got relatively simple microservices, generally just a little webapp or a single function that is automatically connected to a REST frontend. The same concepts apply as you would use for a more traditional web services: usually some mix of remote syslog and ElasticSearch writers.

Networked or remote syslog has been around for a long time and has a fairly robust set of tools around it. You would have to run the central syslog server(s) but the protocol is very simple and there are pure client libraries in every language that you can use for sending logs. One common problem with remote syslog is that it has traditionally been based around UDP. This means that under heavy load, some log messages may be lost. This could be a good thing, helping avoid a cascade overload, but it is something to be aware of. Some newer syslog daemons also support a TCP-based protocol, but client support is less unified so just do your research.

More recent but very popular is logging to ElasticSearch. This is mostly useful because of the Kibana dashboard and Logstash tooklit (often called ELK, ElasticSearch+Logstash+Kibana). Amazon even offers a hosted ElasticSearch option, making it somewhat easier to get started. ES uses a relatively simple REST API, so any language with an HTTP client (read: all of them) should be okay with logging to ES but make sure you are careful with blocking network operations in cases of partial system outages (i.e. make sure your app won't get stuck in a logging call that will never succeed and stop servicing user requests).

More complex logging topologies are bounded only by your imagination, though these days you'll see a lot of use of the Kafka database/queue/whatever-you-want-to-call-it as a nexus point in very complex log distribution systems.

On the "serverless" side, you'll generally want to integrate with these systems directly at the network level, so sending log data directly to syslog or ES from your service/function, rather than writing to local files (though maybe echo to those too for local debugging and development).


Docker-compose containers can't connect to each other

The Dockerfile specifies how a container image is built.  The Dockerfile’s RUN command is trying to ping the db container, which is not yet running during the build phase.
Once the container images have been created, compose will start the containers and they will be able to address each other via their service names (db and web).
To accomplish what you want, an adjustment to your Dockerfile will need to be made.  Replace RUN with CMD.  This will allow the container to execute your ping command when it launches.
FROM alpine
CMD ping -c 1 db

Now your docker-compose up will build the web image, launch the db container and then launch your web container which will successfully execute a ping:
> docker-compose up
Creating network "simple_default" with the default driver
Building web
Step 1/2 : FROM alpine
 ---> a24bb4013296
Step 2/2 : CMD ping -c 1 db
 ---> Running in 0c210f0936fd
Removing intermediate container 0c210f0936fd
 ---> 7c0ed50ad4e8
Successfully built 7c0ed50ad4e8
Successfully tagged simple_web:latest
WARNING: Image for service web was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating simple_db_1 ... done
Creating simple_web_1 ... done
Attaching to simple_db_1, simple_web_1

... output omitted

web_1  | PING db (172.19.0.2): 56 data bytes
web_1  | 64 bytes from 172.19.0.2: seq=0 ttl=64 time=0.218 ms
web_1  |
web_1  | --- db ping statistics ---
web_1  | 1 packets transmitted, 1 packets received, 0% packet loss
web_1  | round-trip min/avg/max = 0.218/0.218/0.218 ms
db_1   |
db_1   | Data page checksums are disabled.
db_1   |
db_1   | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1   | creating subdirectories ... ok
db_1   | selecting dynamic shared memory implementation ... posix
db_1   | selecting default max_connections ... 100
db_1   | selecting default shared_buffers ... 128MB
db_1   | selecting default time zone ... Etc/UTC
db_1   | creating configuration files ... ok
db_1   | running bootstrap script ... ok
db_1   | performing post-bootstrap initialization ... ok
simple_web_1 exited with code 0


Terraform - aws_security_group_rule doesn't support import

Did you try to import in flat setup than modular setup? It seems that in modular setup there is a bug with Terraform. https://github.com/terraform-providers/terraform-provider-aws/issues/559


Use a volume or copy my artefacts into docker image

To copy into the image or to mount volumes--my advice:

if the files will change infrequently, then copy them into the image, with the understanding that if they need to be modified, you will need to rebuild the image.
if you need to frequently change the files, or store some kind of state about the files (and re-use them with future container instances of the image), then mount them as volumes.


How to access local Docker storage on Windows 10?

Most of the persisted data is written to:
C:\ProgramData\docker

you should have directories here such as:

.\windowsfilter
.\volumes
.\config
.\image

etc

The layer and image data is stored under these.


This one from medium should be fine for your purposes: https://medium.com/@ychetankumarsarma/how-to-manage-angular-application-configurations-for-different-environments-with-docker-and-8ff8c55a1c7d.

It is good because:

It shows a test / development scenario.
It notes that it is a good / accepted technique to use environment variables to drive config.
It shows you how to create a config map with a literal value (as an example).
It shows you how to read that environment variable into your app environment properly.

There is also a working sample app noted here -> https://github.com/chetanku/frontend-env which you should be able to "just start up" and work backwards from.


Azure Release Pipeline Fails Using Expand-Archive Command

This was answered via StackOverflow by Kevin Lu:
https://stackoverflow.com/a/66739852/1224614
In short, the second suggestion he made to update to PowerShell 7 and adding the environment information worked. I also had to run the script to reregister the Azure agent on the target server.


Some articles, including this one give some arguments why docker is popular:

  In a nutshell, here's what Docker can do for you: It can get more
  applications running on the same hardware than other technologies; it
  makes it easy for developers to quickly create ready-to-run
  containered applications; and it makes managing and deploying
  applications much easier. Put it all together and I can see why Docker
  is riding the hype cycle as fast as I can recall ever seeing an
  enterprise technology go.

  Moreover, for once the reality is living up to the hype. Frankly, I
  can't think of a single company of any size that's not at least
  looking into moving their server applications to containers in general
  and Docker in specific.

My personal view is that a tool could be successful if it will solve a (major) problem. For example, a half year ago I gave a presentation of docker in the company, the developers had some doubts until they experienced a issue that it was not possible to run all microservices on their local machine.

The same is applicable to the question:

  Are NixOS and GuixSD the incoming Linux distros of DevOps?

It depends whether these distributions will solve a (huge) problem.


Which Jenkins file is responsible for Enable cache setting

Add the following lines in /var/lib/jenkins/config.xml under securityRealm and then restart jenkins service to enable cache. Remove the below 4 line if you don't want to enable it.

<cache>
  <size>500</size>
  <ttl>3600</ttl>
</cache>

Please refer the below image


hello world with flask and redis using docker networking

Yes, it's absolutely reasonable and this is how real apps work. This is the microservice approach which allows to independently scale instances of a service and a DB.
What you have to do is to run your Flask and Redis services in a way so the first one can achieve the second one. If they are on the same machine, you just have to open Redis port used by your Flask app (probably 6379):
docker run -d -p 192.168.10.20:6379:6379 your_redis_image

And specify IP address and port of the Redis container in your Flask app. This is better to be done with a config file like the following yaml:
redis:
  host: '172.16.10.20'
  pw: 'YourPassWord'
  port: 6379
  db: 0

And use it in the code:
import yaml, redis

cfg_file open('config.yaml', 'r')
cfg = yaml.load(cfg_file)
redcfg = cfg['redis']

r = redis.Redis(
    host=redcfg['host'], port=redcfg['port'],
    db=redcfg['db'], password=redcfg['pw'],
)

Don't forget to copy this file into container to be accessible from the inside.


How to stop VPN connection started from Jenkins?

Your question is sitting unanswered here because you're way "out in the weeds" - running pretty idiosyncratic batch scripts that seem to leverage cygwin on Windows, in short, you're in the "wtf" category here.  That's ok!  In this work, all of us are in this place for a lot of our day.

I suggest you consider switching these up for powershell tasks, you can use:

node {
    powershell 'get-process "Openvpn" | stop-process -force'
}

to help with this.  This will simplify your build server and remove external dependencies on cygwin- which if you've ever grappled with at any scale, is a real monster.  It's tough to install automatically, tough to keep up to date, a vuln factory, and poorly maintained.

Perhaps even bigger picture though, using a VPN tunnel in a deployment process is slightly unorthodox.  You might consider pushing an artifact to a repository that both Jenkins and your target server have access to, then polling from the target server for the presence of a new build, then installing it.  Or simply decouple your build and deploy phases with a manual step- one where you log into your production environment to kick off a deployment after a build/test succeeds.


In Docker, is there any advantage or disadvantage in using 'CMD cmd1 && cmd2' versus 'CMD cmd1 && exec cmd2'?

When you replace pid 1 with a new executable, rather than forking it as a child pid, you reduce overhead (slightly). More importantly, you get signals directly to your application instead of having them passed to the shell running as pid 1. By default, the shell running in pid 1 will ignore the SIGTERM sent by docker to gracefully stop your application. After 10 seconds of no response, docker will send a SIGKILL which will immediately terminate the shell and child processes without giving them a chance to drain network connections or finish writing to any files. This can result in things like database corruption depending on what the app is inside the container.

Even when this doesn't apply to your application (if there's no state), you can save up to 10 seconds every time you stop the container by switching to the exec syntax (assuming your application exits when it receives a SIGTERM). Therefore the exec syntax is the one I prefer for both data safety and speed.


Tool to produce nice output about desired vs. current state

I found Pester which is just a powershell unit testing that can save Nunit XML results which can be published inside the pipeline.
It's not "desired"/"current" state I had hoped for but it can in the end provide a nice overview of all the "things" whose states I need to observe.


I would keep the ECS container instances (I'm talking about the Docker hosts - I don't like AWS terminology here) and the deployment as two separate things.

Get your ECS stack up and running. You can manage it through CloudFormation and Auto-scaling groups, that's fine. Just think of your cluster as a platform where you will deploy to, not something you need to redeploy.

Then, for CD, the easiest method by far is to update the service definition to use a new task definition and let ECS rolling update the containers for you.

Every time it start a task, ECS will run docker pull image:tag even if it has the image locally to make sure it has the latest version of that image:tag. So the image tag you use really don't matter (there is no need to change the tag on every build).

That means that you can build myimage:latest over and over in order to deploy it easily.

What you need is a task definition where the image = myimage:latest. Create a service with that task definition and every time that ECS start a task (an instance of your service) it will be the most recent "myimage:latest" you have built.

From there, you are missing only one piece in the puzzle, from CodeDeploy, you can call something, perhaps a lambda function, to create a new revision of your task definition and update your service and ECS will automatically create new tasks for that revision and remove the old tasks.

An example:

Let's assume you have created a service called MyService. That you have configured that service to run 2 tasks for the task definition MyTaskDefinition:1 (revision 1). In that task definition, you have one container definition which image is set to "myimage:latest".

Yesterday you have build myimage:latest which had the ID (SHA) 365d8f7bf565.
Your container instance ABC is running a task named MyTaskDefinition-1-containerName-someLongId. when you inspect that container, it's running the image "sha256:365d8f7bf565.........."
Your other container instance DEF is running another task. It has a similar name (only the ID differ), but it's running the same image.
You push a change to your repo.
CodePipeline picks up that change, build and publish the image to ECR.
That new Docker image is also myimage:latest, but its ID (SHA) is f7ec5e54ac96
Now you need to add a step to your pipeline to use Lambda functions and the AWS NodeJS SDK to do some calls to your cluster:

Create a new task definition (which will be exactly the same as before). That will be MyTaskDefinition:2
Update your MyService to use MyTaskDefinition:2 (instead of 1)

ECS will create new tasks. The container names will MyTaskDefinition-2-containerName-someLongId. When you inspect those containers, you'll see that they will be running "sha256:f7ec5e54ac96.......". Perhaps you'll have 2 tasks on the container instance ABC, perhaps they will be sprayed out (that depends on your service's configuration)
After some time ECS will remove the old task MyTaskDefinition-1-containerName-someLongId from ABC and DEF.

Note: you don't actually need to create a new task definition. If you want you could instead retrieve the service's tasks list and manually stop them one by one. You should wait for ECS to restart a task before stoping a new one (that is: stop the first container, wait for ECS to replace it, stop the second container). When ECS restart the container, it will grab the most recent myimage:latest built, as explained before. I just think creating a new task definition is easier and less error prone (no logic required to wait and check, ECS will handle the rolling update for you if you have a new task definition).


unable to login using docker registry on jenkins

I think what you are doing wrong is the way you specify the credentialID to be used.

Instead of 'awsId' try to use 'ecr:ap-southeast-1:awsId'

docker.withRegistry('https://01234.dkr.ecr.ap-southeast-1.amazonaws.com/', 'ecr:ap-southeast-1:awsId') {
    def image = docker.image("01234.dkr.ecr.ap-southeast-1.amazonaws.com/tas/master-server:${env.BUILD_ID}")
    image.push()
}

EDIT: Check the plugin doc


How to build Java project on existing virtual machine in Azure?

You can use a custom build container image for your pipeline with Maven and all of your other dependencies. You will need to:

Create a Dockerfile that installs Maven and all other dependencies that you have
Build the image and upload it to a custom Azure Container Registry
Modify your YAML pipeline file to reference the new build image

Sample:

resources:
   containers:
     - container: build_container
       image: demoxyz.azurecr.io/pipeline-build-image:latest
       endpoint: AzureCR
       options: '-v /usr/bin/docker:/usr/bin/docker -v /var/run/docker.sock:/var/run/docker.sock'

Here is an article that discusses the topic.


Is there any other approach for tagging git branch using pipeline without credentials in pipeline code?

I believe that what you need is just a Personal Access Token, or if you're a maintainer, possibly a Deploy Token. It's much safer than embedding user credentials.


How to configure a pull request quorum making an organizational constraint?

Although I like the idea, based on what I know while working with Gitlab, Bitbucket and Github I do not believe that it is currently possible as I cannot remember that such integrations were available.

Also Googling: github bitbucket pull request approvals integrated did not return any positive results.

I think that the users should always be part of the same organization in order to approve Pull Requests. However, when I think about github I can remember that I was able to invite people as a collaborator even when they were not part or part of any organization.

In conclusion, I think that it is possible, but only in a certain git-server providers like GitHub. My suggestion is to try it and otherwise report an issue in the associate Github project.


Creating ansible host_vars and group_vars files dynamically

The most elegant way would be to use dynamic inventory, rather than static inventory and dynamic group variables.

However, if for some reason you don't want to do that, one other option is to use custom facts; these are scripts that live in /etc/ansible/facts.d on the client machines.  Like dynamic inventory, they be executable scripts (and therefore, query redis or whatever else), but in this case they define variables just for that particular host.  This isn't quite what you were looking for, since you'd have to query the database on each individual server instead of once for the entire group, but it can have a similar effect.


How to set 'image tag mutability' using CloudFormation?

The ImageTagMutability property in ECR is not currently supported in CloudFormation, as verified by experimentation (nor is it part of the LifecyclePolicy property). There is sometimes lag between feature availability in CloudFormation, and the mutability flag was only introduced in late July.

This comes up elsewhere in the corners of AWS, including:

AWS Global Accelerator - this entire service is not provisionable using CloudFormation.
AWS WAF geo match rules cannot be created in CFN.


Can I persist a Yarn install through different stages in a GitLab pipeline?

We install things in the before_script section of the .gitlab-ci.yml that way on every run of the runner we know what packages exist for each stage.


Can a kubernetes pod be forced to stay alive after its main command fails?

Keeping your pod alive after its main process exited is an anti-pattern and not possible in K8s.

The entire concept of a K8s Job is to run till completion. So what you are looking to do is neither ideal nor recommended.

It would be better to change the entrypoint command of your container to include a retry/error handling mechanism in case of failure.

With that said, you do have ways in which you can debug the failed pod.

If your Pod's container process exits due to error, the Pod will be marked with Status: Failed

But you can still debug the failed pod by attaching an Ephemeral container: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#debugging-with-ephemeral-debug-container

Note: Ephemeral containers are available as K8s alpha feature only from v1.16


What approach should I take to store environment variables for a repo?

General Notes

I don't really understand this one:

  However, this seems clunky, as I have to create the logic to read this
  file in  every script that is made.

You can source all variables from a bash file in one line (source file-name.sh).  If they are "exported", you can pick them up in applications in any language you like (Java, python, or whatever), and most have built in syntax for doing it (like spring boot's SPEL syntax which lets you define your config to directly populate from environment variables automatically on start-up).

You can also make this export part of your ~/.bashrc file or whatever so it's auto sourced in your shell contexts, etc.

Repositories + CI/CD

This may depend on your repository and CI implementation more than the installed code.  For example, GitLab CI and GitHub Actions both support environment variables, as does Jenkins, etc.

In GitLab you can even go to your group level (above a bunch of related projects), and set per-environment environment variables there which are pushed into your deployment so you can use them in your installed code.  Then, you don't even need to store them in any extra files.

References:

https://docs.gitlab.com/ee/ci/variables/
https://help.github.com/en/actions/configuring-and-managing-workflows/using-environment-variables

Docker

Whatever your code is, you can probably bundle it into docker and pass the variables to the container at run time.


How to pick a right hardware configuration setup(server) for MERN stack

Since this is for your learning, it shouldn't tax your hardware incredibly. If you are hosting I would with a free student account if you can. But if you have a linux laptop all of this should work well on it. You're not writing apps that are getting 10,000's of connections and transforming complex data. If you had an enterprise app I would indeed split off the DB from the frontend etc.


What is the difference between modules and workspaces in Terraform

I think a key difference between Terraform Modules and Workspaces is that modules can be inherited and used by other modules and configurations. Terraform Workspaces are intended to be a collection of configurations that represent a single environment, whereas Modules are components that can be utilized by one or more modules/configurations. I think you could use Modules in a similar way to how you are using Workspaces, but you would be contradicting the intent of the Modules.

For example, say that you want to create a base configuration file for all your company's EC2 instances that allows traffic over SSH. You would create a module that spins up a VM, modifies your firewall to allow SSH, and any other configuration settings that you would want all of your instances to have. You would then have every VM Terraform config inherit this file. This would minimize the amount of code that you would have to write and allow for any changes that you would want across all of your TF configs to be made in one central module.


How to create databases, users and grants with terraform mysql and output the created credentials?

To ensure separation and use in other environments, make sure to have an outputs file and set each password as an output.  TF Output Docs.  I haven't done much with for each as it was just released last time I used terraform.  I would imagine a way to iterate over passwords and assign them as outputs.

once you have the output set for the module, I would then have my consuming terraform put those values in a password vault of some kind.  You can use Azure's Key Vault.  TF Azure Key Vault, or any other solution.  Just ensure that the ops team has access to the destination.  This way there isn't anything introduced between when/where the keys are generated and them being stored, reducing the chance for human error.


How to deploy a desktop application using Azure DevOps?

So you want to deploy a desktop application to a Windows machine? Looking through the rest of the documentation, the following links may be helpful:

Windows Machine File Copy task
Build, test, and deploy .NET Core Apps

I can't tell for sure, but you might just be looking for a way for users to download the build artifacts. If that's the case, this link on Build Artifacts may help.


Docker swarm stop spin up containers at 250

I just figure it out. The problem is not with the service or the swarm, it's with the network.

When I use driver: overlay the default subnet is 10.0.0.0/24 which result in 254 address. So I change the mask in the subnet, to 22, which result in 1022 address, I added:

ipam:
  config:
    -subnet: 10.0.0.0/22

And now the network section in the docker-compose file looks like this:

networks:
  web:
    driver: overlay
    ipam:
      config:
        - subnet: 10.0.0.0/22


How to pass a String parameter with spaces in Jenkins to environment variable

If the input string is failing due to folders with spaces not having quotes around them, try adding quotes around every object.
The InpFolderList already adds end quotes:
InpFoldersList="\"$inpFoldersList"\"
Add the quotes around every object.
groovy:
class Example {
   static void main(String[] args) {
        String FolderList = "test1,test2,test3,Folder name1,Folder name2,Folder name3";
        println(FolderList.replaceAll(",","\",\""));
   }
}

Output: (End quotes are left off)
test1","test2","test3","Folder name1","Folder name2","Folder name3
Test Code: https://www.jdoodle.com/execute-groovy-online/ (Copy / Paste Code and Execute)
Using above as an example, you could also add the end quotes and have a complete String variable to pass as $InpFoldersList, which would remove confusion if attempting to debug.
Reference: http://docs.groovy-lang.org/latest/html/api/org/codehaus/groovy/runtime/StringGroovyMethods.html#replaceAll(java.lang.CharSequence,java.util.regex.Pattern,java.lang.CharSequence)


I assume your vault would be being accessed over a network, so you would ask a colleague to use their own command line on their own workstation to enter their key. Mine requires 3 keys to unseal so I just have 3 users independently enter the vault unseal command with their key.


Are manual changes to an aws IAM policy detected by Terraform plan?

Yes, Terraform will detect drift in IAM policies.
Here is my starting point, a policy that allows full S3 and SQS access to specific resources.
data "aws_iam_policy_document" "task" {
  statement {
    effect  = "Allow"
    actions = ["sqs:*"]
    resources = [
      aws_sqs_queue.my-queue.arn
    ]
  }

  statement {
    effect  = "Allow"
    actions = ["s3:*"]
    resources = [
      "${aws_s3_bucket.my_bucket.arn}",
      "${aws_s3_bucket.my_bucket.arn}/*",
    ]
  }
}

resource "aws_iam_policy" "task_role" {
  name   = "my-task-policy"
  policy = data.aws_iam_policy_document.task.json
}

I then manually added ec2:DescribeInstances to this policy via the AWS management console.
To test whether it was detected or not, I ran terraform apply -target aws_iam_policy.task_role
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_iam_policy.task_role will be updated in-place
  ~ resource "aws_iam_policy" "task_role" {
        arn    = "arn:aws:iam::111111111111:policy/my-task-policy"
        id     = "arn:aws:iam::111111111111:policy/my-task-policy"
        name   = "my-task-policy"
        path   = "/"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                  ... REDACTED...
                  - {
                      - Action   = "ec2:DescribeInstances"
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = "VisualEditor0"
                    },
                ]
                Version   = "2012-10-17"
            }
        )
    }

Plan: 0 to add, 1 to change, 0 to destroy.

So Terraform correctly detected the drift.


Custom metric for a Windows process on a Windows ec2 instance

Memory consumption is a metric that has to be reported from inside the guest (EC2 instance).  You can use a CloudWatch agent to collect metrics from the guest directly. The agent also supports custom metrics now.

Alternatively, install a third party monitoring system. Which also means that you'll have to get an agent running on the EC2 instance. There are many of those, choose whatever fits your environment.


Does Docker images works without client application

When you download mariadb image, mysql client tool is only available inside container. It is not getting magically installed on your local - that is why you had to install mysql client app.

Alternative way to connect to db without installing mysql app is to use docker exec and through this get to mysql tool available on container. So you would do something like:

docker exec -it your_maria_db_container_id mysql -h localhost -u root -p

Hope this helps.


Is \u the only way to output the user from .bashrc?

With the release of macOS Catalina (10.15) Apple has switched to Zsh as default shell (instead of Bash).
Instead of setting PS1 in Bash you can set PROMPT in Zsh:
PROMPT="${orange}%n"


Enabling CSRF Protection prevents editing/adding parameters to jobs - intentional or bug?

I believe this may be a result of running Jenkins behind a proxy, which can cause legitimate requests to perhaps appear to Jenkins as cross-site requests.  From the official wiki:

  If you are using nginx as a reverse proxy in front of Jenkins, you need an extra system property on Jenkins "-Dhudson.security.csrf.requestfield=Jenkins-Crumb". See JENKINS-23793 for more details


AFAIK, Swarm does zero real-time resource monitoring. It's only as smart as reservations and limits that you set on the Service beforehand.
First, it takes into account if the tasks of that Service are spread out. It will try not to schedule multiple tasks on the same node, and may ignore total task count per node to do so.
Then, it counts the number of tasks per node and finds the one with the least task count.
All these is restricted by resource reservations. If that node doesn't have the room (based only on the reservations you set), it'll look to the next lower task count.
This is also why Swarm alone can't do built-in auto-scaling because it has no way to track metrics over time.
It does know total CPU and memory, so it knows how to track "full" once you've set resource reservations.
More info: https://github.com/docker/swarmkit/blob/master/design/scheduler.md


Is SSL/HTTPS Needed Between Docker Containers on the Same Network?

Generally - no, I don't think you need TLS between your reverse proxy and service containers within same Docker network (or possibly 2 Docker networks: 1 internal for workers, 1 with internet access where reverse proxy sits and bridges the gap into internal network).
A wider answer would primarily depend on whether you trust your Docker host and services on it. Then consider how containers address each other, do they need to leave docker network (like public name resolution) at any time. And what you'd be looking for from TLS: identity verification or transport encryption. Each of those considerations would have some caveats.


Use Step Functions. They allow you to coordinate lambda functions and have an inbuilt wait state


How to transition from a complex branching reality to a single-branch model?

Because you mention waterfall, I understand that the numerous branches you are alluding to are feature-branches rather than maintenance-branches.

In this setup, I also assume that these branches are created according to a waterfall plan that tries to minimise conflicts.  This implies that the goal of the development is to produce several distinct products.  When using a single-branch development model, it is important to also work on a single product. If several products are simultaneously developed in a single-branch development model, it effectively “glues” together versions of theses products, so that we might have in version a of the repository a healthy product X and a buggy product Y, while in version b the product X has a regression and Y a bug fix, but we have no version where both X and Y are healthy.  Such a situation would force us to consider X and Y as being developed in distinct repositories, which is a hint that they should be.

Therefore, the first steps should perform a repository split:

Arrange the repository so that it is easy to split it into several small repositories. For instance, rearrange the current repository so that each top-level directory corresponds to a repository you want to create in the future. Doing so, you can continue to use the branch-spaghetti discipline everybody is familiar with.
When step 1 is completed, refine the branch-spaghetti discipline by requiring that any single branch can only touch files in one top-level directory.
When each branch complies with step 2, perform the split. Developers can easily convert their pending changes to patch a single repository, just by removing the first-level of the path.

Now that the split has been performed, you can start working on the branch discipline itself.

Introduce programming techniques helping the development of short-lived branches. Branches being short-lived is a crucial aspect of all single-branch methodologies. One of their goals is to reduce the time spent on merging and debugging long-lived branches. A popular technique is the introduction of “feature-flags” where a “factory” uses a configuration flag to either produce the historic version of an object or the new, initially partially developed, version of that object.
By now, you have zillions of repositories with only a few branches in each, and you can turn the “we globally adopt the trunk-based development discipline” button, without seeing the original branch-spaghetti mountain collapsing on the trunk.

The actual split of repositories might be optional, but you would then have to adopt policies that cleanly delineate the allowed scope of each submitted patch (to limit the risk of conflicts when merging changes in the main branch). Reducing the overhead bound to conflicts is one of the goals of single-branch model methodologies, so I assume this is relevant in your context.


How to check application responding in Kubernetes

Kubernetes supports TCP, HTTP, and shell-exec probes for an application startup, readiness, and liveness health checks. If your application is a web application, you can use HTTP based liveness healthcheck as shown in the following code snippet:
apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-http
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/liveness
    args:
    - /server
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 3
      periodSeconds: 300


User Data in launch configuration of amazon web services

Make sure the EC2 instance has permissions to create the volumes. See https://aws.nz/best-practice/ec2-instance-roles/
Check for the errors in /var/log/cloud-init-output.log and (much more verbose) /var/log/cloud-init.log
Be aware that you can't attach volumes from other AZs to your EC2, so not quite sure why you would want to create them from the launch script.
(added based on comments) Specify region: aws --region us-east-1 ec2 create-volume ...

Hope that helps :)


AWS container service team member here. IF you are still early on in your journey to automate this via a CLI I would suggest you consider moving past the ecs-cli. The follow-on version of this CLI is called AWS Copilot and has more sophistication to implement those workflows. Note that Copilot dropped support for docker compose because we have implemented a native integration with Docker and the new docker compose cli. You can read more here and here.


AWS provide different services and categories them on base of computation, storage, databases, management, Analytics, Messaging, Developer Tools etc.

ECS comes under the computing services.

Amazon EC2 Container Service (ECS) – Container Management for the AWS Cloud

  Run your applications packaged as Docker containers.EC2 Container Service (or ECS for short). This service will make it easy for you to run any number of Docker containers across a managed cluster of Amazon Elastic Compute Cloud (EC2) instances using powerful APIs and other tools. You do not have to install cluster management software, purchase and maintain the cluster hardware, or match your hardware inventory to your software needs when you use ECS.

Here I will explain the steps to set up the docker based application in ECS and terminology.

Cluster: A cluster is a pool of EC2 instances in a particular AWS Region, all managed by ECS. One cluster can contain multiple instance types and sizes, and can reside within one or more Availability Zones.

This is the first step to creating ECS.before anything you need cluster it can be default or you can create new one.

EC2 Instance type: In cluster you need to choose the Amazon EC2 instance type for your container instances. The instance type that you select determines the resources available for your tasks to run on.

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud.it's like your PC you can install mongo, MySQL (for MySQL RDS is best), PHP, node or whatever you want in it.While Creating EC2 you have to define which OS you want to use in ec2.You can access EC2 using pem file  that will be provided by AWS during creating ec2.https://aws.amazon.com/ec2/

Container: – A container is a packaged (or “Dockerized,” as the cool kids like to say) application component. Each EC2 instance in a cluster can serve as a host to one or more containers.
For every task definition, you need to specify container which tells about CPU, memory and other resources.
ECR Registries Amazon EC2 Container Registry (Amazon ECR) is a managed AWS Docker registry service that is secure, scalable, and reliable. Amazon ECR supports private Docker repositories with resource-based permissions using AWS IAM so that specific users or Amazon EC2 instances can access repositories and images. Developers can use the Docker CLI to push, pull, and manage images.

Once the above environment is ready you can place task or start services.

Task definition: A task definition is like a blueprint for your application. In this step, you will specify a task definition so Amazon ECS knows which Docker image to use for containers, how many containers to use in the task, and the resource allocation for each container.
service services manage the task that you have created.once you have created a task definition, you will configure the Amazon ECS service. A service launches and maintains copies of the task definition in your cluster. For example, by running an application as a service, Amazon ECS will auto-recover any stopped tasks and maintain the number of copies you specify.

NOTE: Elastic load Balancing can be setup before any thing done. when you want to create services you can configure with load balancer.load balancer conttain target group and container which need to be balanced.https://aws.amazon.com/elasticloadbalancing/

Elastic Load Balancing Elastic Load Balancing automatically distributes incoming application traffic across multiple Amazon EC2 instances. It enables you to achieve fault tolerance in your applications, seamlessly providing the required amount of load balancing capacity needed to route application traffic


Unable to get value of password parameter into another parameter

Could credentials binding plugin be the answer?
The idea is to add svn user and password to Jenkins credentials store and then use it in the pipeline.
withCredentials([
    usernamePassword(
        credentialsId: 'mySvnLogin',
        usernameVariable: 'USER',
        passwordVariable: 'PASSWORD')]) {
    sh '''
      svn ls <url> --non-interactive --no-auth-cache --trust-server-cert --username $USER
 --password $PASSWORD
    ''''


Will Docker Swarm work if masters are spread over 3 continents?

There are various timeouts within swarm mode that do not handle high latency well and may start to assume nodes within the cluster are down, lose quorum, or have networking issues. For HA, the standard advice is to have multi-AZ but not multi-region for a single cluster. For use cases that need multi-region, you would deploy multiple swarm clusters, one per region. If you really need multi-region with a single swarm cluster, then consider placing all of the management nodes within the same region and only deploying worker nodes to the other regions.


I have not tried it but I think if you change your deny to a list of elements it will work.
{
  "Effect": "Deny",
  "Principal": {
    "AWS": "arn:aws:iam::<redacted>:root"
  },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringNotLike": {
      "sts:RoleSessionName": [
        "${aws:username}",
        "AnotherAllowedName"
      ]
    }
  }
}

This will allow an IAM user to assume the role if they use their username as the session name and it will also allow other Principals to assume the role if they use "AnotherAllowedName" as the session name but all other session names would be declined. I found this AWS blog post helpful when researching this answer


If a piece of work requires manual testing/QA, do we need to wait until opening PR?

The guidance that we've always followed in my various scrum teams is developers should get what they've got into testing as soon as possible. It may mean a little rework later on down the line if there are issues found in code reviews or developers haven't completely finished coding, but the earlier developers get feedback on what's been done, the earlier they can fix any issues, and take account of this feedback in any further coding.

You should make testers aware when they aren't getting completely finished output, but if they've been encouraged to think about the benefits of preventing bugs as well as just finding them, they should hopefully be keen to adopt this process.

It isn't always practicable - for example, if you don't have deployment tools that are flexible in which branches they deploy from, but when it is practicable, it's what Scrum and agile are all about - get feedback and get it early.


any way to limit "docker logs" output by default?

One way to limit docker logs is to specify limits in docker daemon's config file.
You can put this in /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

and then restart your docker daemon.


Create pull request for docker dependency package if outdated

Assuming a docker file like this...

FROM ubuntu:18.04

ARG TF_VERSION=0.12.25

RUN apt-get install wget -y

RUN wget -O terraform.zip https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip

In your GitHub action you would need to get the current version from your Dockerfile.

export CURRENT_TF_VERSION=$(grep -Eo '[[:digit:]]*\.[[:digit:]]*\.[[:digit:]]*' Dockerfile)

Then the latest version of terraform.

export LATEST_TF_VERSION=$(curl -s 'https://checkpoint-api.hashicorp.com/v1/check/terraform' | jq -r .current_version)

Now in bash or using if and outputs in GitHub actions compare the two values. If they are not the same then find and replace the current version with the latest version in your Dockerfile.

sed -i "s/${CURRENT_TF_VERSION}/${LATEST_TF_VERSION}/" Dockerfile

Then you can use the create-pull-request GitHub action to create a pull request.

The create-pull-request documentation doesn't mention what happens if you don't have any changes but I asked here. If it doesn't throw an error when you don't have changes then you could simply get the latest value and run sed.

sed "s/[[:digit:]]*\.[[:digit:]]*\.[[:digit:]]*/${LATEST_TF_VERSION}/" Dockerfile


In Jenkins, how can parameters that contain spaces be referenced?

You can use params['IP Address'].

Think of params as a Map containing a key 'IP Address'. If the key didn't have a space, then you could use params.IPAddress or params['IPAddress'], but when there's a space, you can only use the latter syntax.

That syntax is also useful when you're computing the key name to look up, and have stored it in a variable:

final String myParam = "param${i}"
// can't say `params.myParam`
echo "Value is ${params[myParam]}"


How to check return status of parallel branches in jenkins pipeline

You may want to try the Join plugin. Some more details about this plugin (from the linked page):

  This plugin allows a job to be run after all the immediate downstream jobs have completed. In this way, the execution can branch out and perform many steps in parallel, and then run a final aggregation step just once after all the parallel work is finished.

  The plugin is useful for creating a 'diamond' shape project dependency.   This means there is a single parent job that starts several downstream jobs. Once those jobs are finished, a single aggregation job runs.

For more details on that, refer to my self answered question about "Why trigger something remotely (using Jenkins) and then just forget about it?".


What's the practice to perform a releasing with Git?

Consider to substitute the version in POM by the git describe upon build without hardcoding the version in your repository, you could do this with e.g. sed from below version_slug.

export git_version=`git describe --tags --always`
export version_slug="$(sed s/-/\./g <<<$git_version)"

and then substitute the version_slug in pom.

In that way for each commit and/or tag in gitlab, you'll create a new version, with out actually changing the repository.

this is what i did for a ruby project: jekyll-plantuml-url, where the version is not coded in the source, but is set upon build, and after testing deployed as the git tag.

  script:
   - bundle install
   - rake set_git_tag
   - rake
   - gem build jekyll-plantuml-url.gemspec

Furthermore don't understand the need of all those branches, with CI/CD we continuously build, test and deploy, that means before accepting a merge to master you have a deployment for QA to do the testing, code review, functional testing ...etc.

Next add QA as approver for a merge to master, thus make it mandatory that QA signs of a merge request to master and integrate quality in the development cycle or have a master that is always ready for release.

One could opt to work from master and for a release tag the master branch with a version tag, as per above example.

OR

like gitlab does it, to create a branch for each release, cherry pick the features from master that are ready or planned and then tag the branch with a version tag.

where as a version tag creates a deployment for a commit, release or a RC.


How do I create multiple versions of a lambda function in terraform

This is shown in the Hashicorp tutorial - "a new version of the lambda function. The suggestion there is to point to a different zipfile in S3.

I suppose in your case, the filename either needs to change, or the file contents need to change.

The suggestion from the tutorial is to use a variable for the filename, associated with the version of the application.


Shallow clone

You can indeed get a shallow clone out of Git using:

git clone --depth=1 <url>

This will still clone the repo and create a .git folder with the objects, only smaller in size (difference depending on your total file size vs. history size).

Git archive

You can also use git-archive to extract an archive of the repo:

  Creates an archive of the specified format containing the tree structure for the named tree, and writes it out to the standard output. If  is specified it is prepended to the filenames in the archive.

In the examples it shows for instance:

  git archive --format=tar --prefix=git-1.4.0/ v1.4.0 | gzip >git-1.4.0.tar.gz

    Create a compressed tarball for v1.4.0 release.

Hosted Git, archive API

If you are hosting your repo on GitHub, then you can use their archive API:

https://api.github.com/repos/<username>/<repository>/zipball/<commit_hash>

Bitbucket.org has a same functionality for this:

https://bitbucket.org/<username>/<repository>/get/<branch_name|commit_hash|tag>.zip


What is the proper term for lead time from commit to deployment?

From code repository to any environment, as a running software would be "Time to Deploy" ( which is different from deployment time, btw ).

From code repository all the way to production environment, the best aproximation is time to market (TTM), taking aside that time to market does involve the conceptual design of a product.

But as you said you are tagging an pipeline, I should advise:

Use the tags that makes sense: to you, your team and to your product.

A common pipeline could be : BUILD > TEST > DEPLOY

But we could tag the same pipeline, making more steps visible, ie :
SOURCE CHECKOUT > CODE CHECK > UNITARY TESTS > BUILD > INTEGRATED TESTS > DEPLOY > ACCEPTANCE TESTS

As a rule, if you have trouble on one step and must check the output to understand whats going on, this step could be divided in one or more tags.

If makes sense to you, you could even tag something like SOURCE CHECKOUT IDLE TIME ( the time took for the build machine to connect to the repository server, before downloading the source )

Hope this could be useful, happy tagging!


Reg uploading additional files in docker hub

Short answer: No.

Long answer: Docker Hub will keep the image. Anything else associated with the image needs to be kept in the source repository linked to the image in the registry.


Jenkins shows "No changes in any of the builds."

Temporary workaround

Instead of calculating changes between builds compare changes against specific branch.

Open job configuration
Go to Source Code Management -> Git -> Additional Behaviours
Add Calculate changelog against a specific branch
Enter name of repository (origin in my case), and branch (master in my case)

Now instead of showing only new changes, each build will show all commits in release branch that are different comparing to master.


private subnets in an Auto Scaling Group on aws?

There is a range of common scenarios when you want to use private subnets to be used in an auto scaling group:

Your traffic is terminated by reaches your infrastructure on a  Elastic Load Balancers and your web server instances are behind the load balancer. This reduces the attack surface of the web servers.

( I think I started to use "terminate" since seeing it being used for SSL/TLS termination, ie the server(s) that hold the secure certificates. But because you could also use a Network Load Balancer at the "edge" of your infrastructure, I see now its not appropriate to use in this context)

Your auto scaled resources are actually not responding to outside traffic, for example if they are worker resources processing messages from an SQS queue.

There was a similar mention of both "public and private subnets" all over my LinuxAcademy exam preparation, depending on your thinking processes it may be straightforward to just say: "autoscaling can use subnets", but then this may confuse other people.


How To Determine Min/Max Resources For Kubernetes Pods?

The DevOps toolkit 2.3 book by Victor Farcic goes into some detail on this.

What You Should Understand

You as a developer specify whatever requests/limits you like initially.
These values have no baring on reality.
In order to get "real" values, you must monitor your application's real resource usage.
This is best done through a monitoring tool like prometheus, instana, datadog etc.
E.g. in instana, I can see that I have 512MB set up for a pod's requests, 2G for its limits, but the pod only has one container currently taking 57MB.
But I can't just use 60MB as a value now... I need to load test my application and see how it behaves under real/stressed circumstances.
Then you review these values and you can adjust the requests/limits accordingly.

Determine Usage Without a Monitoring Tool

You can use some kubectl commands to get these values as a one off (it's annoying at scale).  Here is an example of a different app I have running.

Notice that requests/limits are set to 1/4-2 CPUs and 4G-8G memory.  But the container is really using almost no CPU and 1.198G memory.  If I load tested this and those values did not move up, I would need to change them.  But I have load tested it and know that they would go up towards these realistic requests/limits, so it's okay.

$ kubectl describe pod -n gazelle gazelle-ws-0-0-1-6d67cffcf-lvlhk

...
    Limits:
      cpu:     2
      memory:  8Gi
    Requests:
      cpu:      250m
      memory:   4Gi

$ kubectl top pod gazelle-ws-0-0-1-6d67cffcf-lvlhk -n gazelle --containers

POD                                NAME         CPU(cores)   MEMORY(bytes)
gazelle-ws-0-0-1-6d67cffcf-lvlhk   gazelle-ws   5m           1198Mi


Does QA in DevOps roles means CI/CD or more?

First, DevOps is a journey, not a destination.  Not every organization approaches similar problems with the same solution; while automation is a component of a DevOps platform, it's not the single defining characteristic.

The role and responsibilities of QA professionals haven't really changed; they're still responsible for validating that code is ready for release.  What HAS changed is the focus on both automation and smaller releases.  Much like operational roles, QA staff should now have the opportunity to focus on adding additional automatic quality checks into the development life cycle.

In a shop with a mature CI/CD pipeline, QA becomes more about designing great tests, and letting the software do the actual heavy lifting. As Tensibai commented above, "some QA tests can't be automated" (yet), so there will still need to be some engineering responsibilities, but the goal should be to utilize automation in every aspect of delivery in order to facilitate adding additional quality.


Azure has a robust CI/CD pipeline which is much easier to use than CodeDeploy on AWS. It is called Build/Release pipelines under Azure DevOps. It allows you to configure multiple environments with build and release (deployment) stages for each. It also allows for easy configuration and interoperability with Azure Webapps/VMs and other resources available on Azure. Having recently switched from Azure to AWS, I definitely miss the ease of configuration and the intuitive UI on Azure DevOps.

Azure DevOps integrates a version control repository system, a agile style work items/boards which can also be used to configure your backlogs and iteration items. There is even a dashboard to integrate and monitor your work items and deployments across the team. So it is definitely much bigger than just a devops tool or CodeBuild.


jenkins stuck "Please wait while Jenkins is getting ready to work"

OK, a good week-end does its job as usual: My issue was indeed caused by my company proxy (a good thing would be to add this requirement in the jenkins doc, i might have  missed it anyway).
So the fix is actually simple, just need to add the correct options to the jenkins start command, as in:

-Dhttp.proxyHost=<your_proxy_ip_here> -Dhttp.proxyPort=<your_proxy_port_here> -Dhttps.proxyHost=<your_proxy_ip_here> -Dhttps.proxyPort=<your_proxy_port_here>

Full example:

/usr/bin/java -Djava.net.preferIPv4Stack=true -Dhttp.proxyHost=10.21.92.40 -Dhttp.proxyPort=3128 -Dhttps.proxyHost=10.21.92.40 -Dhttps.proxyPort=3128 -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080

tcpdump was the savior here (+ a colleague :) )


elasticsearch-fluentd helm chart on Openshift 3.11 pod failing

Changing logging driver form journald to json has fixed it.
I edited /etc/sysconfig/docker, and set

OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false'

Then restarted docker on the nodes I needed:

systemctl daemon-reload && systemctl restart docker


How can I share a (global) variable between salt states?

When I encountered a related problem, I wound up having Jinja run execute Salt modules.

In my case, it was cmd.run to run mktmp (and, at the end of the template, rm for cleanup), with file.append and file.grep, though in your case, you might be able to use Salt's load_yaml extension to Jinja.

So, something like this (rough, untested):

{% set tmpfile = salt['cmd.run']('mktmp') -%}
{% salt['file.append']('my_key: ' ~ your_key) -%}
{% set my_key = load_yaml(tmpfile)['my_key'] -%}
{% salt['file.remove'](tmpfile) %}

This certainly isn't the best place to put data, but it demonstrated using calling into Salt modules from Jinja to store data. I don't know of any existing Salt modules suitable for storing ephemeral data that only lives as long as the templates compile. I'll check back later and see what other solutions might work.


creating private registry, id and password with -Bbn option

From docs:

-B  Use bcrypt encryption for passwords. This is currently considered to be very secure.
-b  Use batch mode; i.e., get the password from the command line rather than prompting for it. This option should be used with extreme care, since the password is clearly visible on the command line. For script use see the -i option. Available in 2.4.4 and later.
-n  Display the results on standard output rather than updating a file. This is useful for generating password records acceptable to Apache for inclusion in non-text data stores. This option changes the syntax of the command line, since the passwdfile argument (usually the first one) is omitted. It cannot be combined with the -c option.


Based upon my understanding you essentially have a Data Partitioning problem, in that you need to maintain a degree of separation between each of your customers but also host the service centrally.

You have suggested that you instantiate a Docker container for each of your customers, I think there are a couple of issues with this:

Containers are designed to be somewhat ephemeral, you can use some of the concepts within Kubernetes to effectively create a singleton per customer, however:

Availability will likely be hurt by virtue of it being a singleton if the pod, node or cluster fails then that pod will need to be rescheduled.
The cost is likely to be quite high, as each of those containers will consume some resources regardless of whether they are being used or not.

Depending on how many customers you have, you get into the realms of too many containers:

Orchestrators such as Kubernetes, Docker Swarm and Mesosphere are generally designed to support a large number of containers, however, the expectation is that's lots of the same workload not lots of disparate workloads. Kubernetes has in the past experienced slowdown when lots of ConfigMaps, Pods and CustomResourceDefinitions are deployed.
Whilst lightweight each container will consume system resources, not just Memory, CPU but also TCP/IP pool exhaustion.

Isolation - containers are a good way to isolate workloads from each other, but it's not infallible - Read Containers are not VMs

If I were to approach your problem I would be inclined to use Kubernetes as a Microservices platform hosting your application across multiple ReplicaSets then implement your partitioning at the datastore level in AWS via DynamoDB and EFS.  You are already encrypting your data so you have cryptographic certainty that isolation can be achieved.


@PrestonM indicated that:

  Whenever I see an "unexpected at this time" error it's in my batch
  scripts, and it's either because I didn't use the correct escape
  character, or an absolute path is incorrect. I don't see Anaconda
  anywhere in your PATH, but I do see Xtralien. Is it possible that you
  need to add Anaconda?

OP feedback was:

  Not changing the default directory and adding it my path variable
  during the installation seems to have done the trick. Thanks a bunch
  for the advice!


How do I enter a multi line variable in Terraform?

Terraform variable type constraints separate "primitive" types from "complex" types. These primitive types include strings:

A primitive type is a simple type that isn't made from any other types. All primitive types in Terraform are represented by a type keyword. The available primitive types are:

string: a sequence of Unicode characters representing some text, such as "hello".
number: a numeric value. The number type can represent both whole numbers like 15 and fractional values such as 6.283185.
bool: either true or false. bool values can be used in conditional logic.

So, simply declaring the variable as a string, with the linebreaks in it would suffice.


The git hooks approach could probably work fine for lone developers. But when multiple developers work on the same project it doesn't work so well, especially in large scale projects.

Several reasons:

the git hooks are local to your copy of the repository, there is no check for consistency across all developers. Anyone can "adjust" their hook script as they see fit, or even forget to get them set up (they're not automatically installed when cloning git repositories for example). A CI platform will consistently run the same scripts for all changes.
the verifications performed are typically done on the developer's computer, so you're right into the typical "it worked for me" case. Again - consistency: a CI platform will perform the same verifications, in the same context, on a pool of resources much easier to keep consistent than developers' environments
the verifications performed by a developer are fundamentally silo'd, performed in isolation, on the local snapshot of the repository branch, before the changeset is actually integrated into the branch - they do not take into account the changes from the other developers which can happen at any time. Results obtained that way could be significantly misleading, see How can successfully pre-verified changes cause regressions which should have been caught?. CI platforms focus on the repository branch itself, checking how a particular changeset performs on that branch at that particular moment, with all other changesets already taken into account, reflecting the actual branch integration status.
(in larger projects) a CI tool can perform verifications which are impractical for the individual developers. For example multi-platform build lasting hours and requiring high performance build machines well above the typical developer laptop. The centralized verifications performed by a CI platform can be much more resource frugal than when performed by each individual developer.

Side note: not all projects benefit from the developer ability to spin up on demand a production-like environment/clone server ;) But even if possible, most of the above reasons still stand


First, see Docker daemon’s log. Refer docker docs

If some request or command executed to the Wordpress container, it might had restarted through docker daemon.


How do I configure an AWS network load balancer with an A record in Cloudflare

You create a CNAME record referencing the assigned hostname of the balancer as the target.

CNAME records are not valid at the apex of a zone (i.e. the "naked" domain, without www or any other prefix) but Cloudflare allows you to create one there -- sort of -- they actually allow you to believe that you're creating one there, and then their service automatically transforms it into one or more valid A-records, dynamically.

This feature is enabled by default at the apex.

https://blog.cloudflare.com/introducing-cname-flattening-rfc-compliant-cnames-at-a-domains-root/


What's missing from this simple recipe that creates a user?

I'll put in an answer although I'm not 100% sure. I think that you got the permissions wrong on ~/.ssh and it should be 0700 instead of 0600, otherwise the user cannot access it.

Another tip which is unrelated to your problem, the user resource has a property manage_home (https://docs.chef.io/resource_user.html#properties) which will create the home directory for you and will you save a redundant resource.


Why won't InSpec admit that port 5432 is listening inside a PostgreSQL container?

You did everything right, it is the correct InSpec test and use with Docker!

You encountered a bug in InSpec that is related to InSpec's netstat output parsing combination with Alpine's netstat output.

See InSpec's output for postgres container instead of postgres:alpine:

$ inspec exec docker_test.rb -t docker://1c8162517c22                                                               1d15h master[a5ff79d3]

Profile: tests from docker_test.rb (tests from docker_test.rb)
Version: (not specified)
Target:  docker://1c8162517c229ff8c87ab3e8c909f46b4370d7c63a65c0f973a5c10fdebda800

  Port 5432
     ✔  should be listening

Test Summary: 1 successful, 0 failures, 0 skipped

In your case, the problem is that Apline's netstat reports a different output then other Linux eg. Ubuntu, CentOS:

# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      999        108791     -
tcp6       0      0 :::5432                 :::*                    LISTEN      999        108792     -

On Alpine, the inode output is missing:

netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      -
tcp        0      0 :::5432                 :::*                    LISTEN      -

That is why InSpec is not able to parse the netstat output. It is clearly a bug in InSpec. As a mitigation, I recommend to install the iproute2 package for alpine containers via apk add iproute2. InSpec prefers ss over netstat. Please report this bug to InSpec team.


I bet you're not the only one with such experiences :)

One possible approach today (which wasn't available in the early DevOps days) is to counter-argument with the many research-based publications from industry-recognized influencers. Like Accelerate or 2017 State of DevOps Report to name a few.

Another approach could be to propose and implement DevOps-based pilot projects to actually measure and compare performance parameters against the existing silo'd solutions. This could be tricky, though, as those influencers could insist on requirements cornering the pilot projects. For example insisting on individual/small team development speed which could be an obstacle for optimizing and speeding up the overall larger/entire project team's development.


Your docker image could be without an OS, that's called distroless image.

https://docs.docker.com/develop/develop-images/baseimages/

https://github.com/GoogleContainerTools/distroless


How to see previous version of a Docker image?

docker image inspect - will show available information on the container (docs)

docker image history [your OPTIONS] imagename - Shows history of the container (docs)


How do i connect docker pipeline on jenkins to kubernetes cluster

Maybe it is strange solution, but I would like to provide it to you:

You can get actual kubeconfig file from your Kubernetes cluster or locally and change it to base64 format like in example below:

- cat /root/.kube/config | base 64

Then use your base64 configuration file in pipeline:

- echo $your_variable | base64 -d | tee /root/.kube/config

In such case you will be having actual kubeconfig file on runner machine.

In case of DNS problem - just need to specify IP address of your runner machine in /etc/resolv.conf file.


Kubernetes supports namespaces.

  multiple virtual clusters backed by the same physical cluster

These namespaces can allow you to use one cluster for all your environments. You can also do fancy things in your CI/CD pipeline so that each branch or each commit gets its own namespace. Those pods in prod namespace can still speak to pods in dev namespace unless you segment with NACL's. Just make sure you CNI supports that.

That being said most large companies I have worked for are running a cluster per environment.

I don't think there are right or wrong answers here. Do what you think is best for your company/team/workload/environment etc.


How do I hire a good DevOps, fitting my company?

It is a common misbelief that “DevOps” is a role in a company. The word merely denotes an organisational shift in software companies, this shift can be shortly described by the “You build it, you run it!” coined by Werner Vogels:

There is no need to distinguish between building and running, and according to Werner Vogels, it is much better than that:

Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.

This lesson is interesting, but this particular change in an organization is not always easy to implement.

This shift is concomitant of an increasing importance of continuous integration and deliveries methods. Please note that these ideas are much older than the formalisation of “agile” methodologies, some of them are already present in the NATO Scientific Committee report about software engineering published in … 1968!
While the wording can be overlooked, the short description of the DevOps proposal (this site) mentions “software engineers”:

Beta Q&A site for software Engineers working on automated testing, continuous delivery, service integration and monitoring, and building SDLC infrastructure.

Now if you are looking for a software engineer that can help your team getting started with these methodologies, engage them on this topic. Even if you are not familiar with this topic, following the “You build it! You run it!” slogan you can start from a programming problem and lead the interview towards question about deployment and maintenance of the application.


When to "npm publish" when using git flow

I'd set up my CI system to do this on every commit to master, that is, after the release branch merges.

The first very straightforward reason for this is that the commit to master gets tagged with the release, and you want the thing you ultimately publish to match exactly the version of the source code at the tag.  While there shouldn't be any merge surprises if git flow is run perfectly, it's still better to publish the content of a tag and not the commit immediately before a merge.

The second reason is that it is a very simple rule that can be automated.  If you do any amount of final testing on the release branch, you need to make a manual decision as to whether or not you can release or not; your CI system (probably) can't make that decision just based on the presence of a release branch.  On the other hand, since every commit to master is merging in a release (or hotfix) branch that you've fully validated, the commit itself is a signal that you're ready to release.


Does my organization need adopt Agile Soft. Dev. before adopting DevOps?

Agile Software Development is not required to do DevOps, but I believe the argument can be made that the value proposition for DevOps is often a lot lower without Agile.

DevOps is a lot of things, but automation is a central theme.  The value of automation increases in direct proportion to the frequency with which Development creates new releases.  Frequent deployment has a positive impact on certain types of products, specifically consumer applications.  High velocity all the way through the software delivery lifecycle returns value for each iteration (anyone seen the CA ad where the zombies want new features in their apps?)

Without Agile, high-frequency releases are extremely difficult, if not impossible.  If the Development team is releasing software once a quarter, or twice a year, DevOps can still automate the process, but then what is the point?  The investment in time, training, and resources to adopt DevOps may be partly returned in quality, however, the best value is in maintaining high velocity throughout the delivery lifecycle.

One could also argue that if you're going to adopt DevOps, why wouldn't you also adopt Agile?  The principles that make them both work, work well together.  Practicing DevOps by itself, without Agile, could create an imbalance between Ops and Dev, in which Ops is outperforming Dev for service delivery.


Docker on Windows: Write directly to host's EventLog?

On Windows (Dotnet Core) it can be done using powershell remote-invoke.

In the container you need to set a trusted host by doing:

RUN @powershell Set-Item WSMan:\localhost\Client\TrustedHosts -Value '*' -Force

Ideally replace the * with the host actual IP address. Keep in mind it changes between system restarts.

On the host you need to create an Event Source called MyLogger.

After that in your app you can create a custom logger that you add to the Microsoft Logging pipeline.

RemoteEventLogProvider.cs:

using System.Management.Automation;
using System.Net;
using Microsoft.Extensions.Logging;

namespace LogWriter
{
    public class RemoteEventLogProvider : ILoggerProvider
    {
        private readonly string _computer;
        private readonly PowerShell _powershell;
        private readonly NetworkCredential _credential;

        public RemoteEventLogProvider(string computer, NetworkCredential credential)
        {
            _computer = computer;
            _credential = credential;
            _powershell = PowerShell.Create();
        }

        public ILogger CreateLogger(string category)
        {
            return new RemoteEventLogger(_computer, _credential, _powershell, category);
        }

        public void Dispose()
        {
            _powershell.Dispose();
        }
    }
}

RemoteEventLogger.cs

using System;
using System.Management.Automation;
using System.Net;
using Microsoft.Extensions.Logging;

namespace LogWriter
{
    public class RemoteEventLogger : ILogger
    {
        private readonly string _computer;
        private readonly string _category;
        private readonly PowerShell _powershell;
        private readonly NetworkCredential _credential;

        public RemoteEventLogger(string computer, NetworkCredential credential, PowerShell powershell, string category)
        {
            _computer = computer;
            _category = category;
            _credential = credential;
            _powershell = powershell;
        }

        public IDisposable BeginScope<TState>(TState state)
        {
            throw new NotImplementedException();
        }

        public bool IsEnabled(LogLevel logLevel)
        {
            return true;
        }

        public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter)
        {
            var script = ScriptBlock.Create($"$password = ConvertTo-SecureString '{_credential.Password}' -AsPlainText -Force;" +
                                            $"$credential = New-Object System.Management.Automation.PSCredential ('{_credential.UserName}', $password);" +
                                            $"Invoke-Command -ScriptBlock {{ Write-EventLog -EntryType {logLevel} -EventId 0 -Source MyLogger -LogName Application -Message \"{formatter(state, exception)}\" }} -ComputerName {_computer} -Credential $credential;");

            var results = _powershell.AddScript(script.ToString()).Invoke();

            _powershell.Commands.Clear();
        }
    }
}

Example usage Program.cs:

using System;
using System.Net;
using System.Threading.Tasks;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;

namespace LogWriter
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var config = new ConfigurationBuilder()
                      .AddCommandLine(args)
                      .AddEnvironmentVariables()
                      .Build();

            var computer = config["ConnectionStrings:Host.Address"];
            var provider = new RemoteEventLogProvider(computer, new NetworkCredential($"{computer}\\username", "password"));

            using (var factory = new LoggerFactory().AddConsole())
            {
                factory.AddProvider(provider);

                var logger = factory.CreateLogger("MyLogger");

                logger.LogInformation($"Remote log at {computer}");

                while (true)
                {
                    try
                    {
                        logger.LogInformation($"The time is {DateTime.UtcNow}");

                        if (DateTime.UtcNow.Second % 2 == 0)
                        {
                            throw new Exception("random shit");
                        }
                        else
                        {
                            logger.LogWarning("Example warning");
                        }
                    }
                    catch (Exception ex)
                    {
                        logger.LogError(ex, ex.Message);
                    }

                    await Task.Delay(TimeSpan.FromSeconds(5));
                }
            }
        }
    }
}

Example Dockerfile:

# Build
FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /app
COPY . /app

RUN dotnet publish -c Release -o /app/out

# Runtime
FROM mcr.microsoft.com/windows/servercore:ltsc2019
RUN curl -o c:\dotnet.exe https://download.visualstudio.microsoft.com/download/pr/a9bb6d52-5f3f-4f95-90c2-084c499e4e33/eba3019b555bb9327079a0b1142cc5b2/dotnet-hosting-2.2.6-win.exe
RUN c:\dotnet.exe /quiet /install

# Host
RUN @powershell Set-Item WSMan:\localhost\Client\TrustedHosts -Value '*' -Force

# Application
WORKDIR /app
COPY --from=build-env /app/out .

ENTRYPOINT dotnet LogWriter.dll


Multiple Microservices in CI environment

So if we change it for one service, we need to update it in each docker-compose.yml in each project.

That's sounds like duplicate code and there are many ways to avoid that. But without more detailed information about how your setup looks like it's difficult to help.

Within a pipeline you can access other projects and their source code. Maybe Git Submodules is an option. Or you might be interested in the include keyword of GitLab CI.


How to properly manage mupltiple environmental variables with Gitlab CI and Docker Swarm?

There is stages in gitlab ci and on 'docker build' stage I don't need ALL env variables. Usually its only a REACT_APP_SERVER_URI variable.
Later I use env file in compose.yml like you are.

env_file:
  - ./envs/common.env


Update volume size dynamically from inside a POD

In my opinion, you could create bucket (aws s3 for example) and store your data there (create simple app to make upload), thus, you don’t need to care with disk full error


Enable TLSv1 TLSv1.1 for specific Ingress with Nginx Ingress

The ConfigMap and Annotation configuration have different purposes.
As per nginx-configuration/configmap/#ssl-protocols:

Sets the SSL protocols to use. The default is: TLSv1.2 TLSv1.3
ngx_http_ssl_module.html#ssl_protocols
Context:  http, server

i.e., enable the specified protocols to your Nginx Controller.
client --ssl-protocols-> controller --> backend

On nginx-configuration/annotations/#backend-certificate-authentication, we can read:

Enables the specified protocols for requests to a proxied HTTPS server.
ngx_http_proxy_module.html#proxy_ssl_protocols
http, server, location

i.e., enables the specified protocols for requests to a proxied backend using HTTPS.
client --> controller --proxy-ssl-protocols-> backend

As you can see, to set the protocols served by your Nginx controller, you only have http, server contexts available (i.e., you can't set this to specific ingress objects).


GitLab Runner is using its Docker host's hostname as Git server FQDN/hostname

Neil, it's because the = sign in your /etc/gitlab/gitlab.rb file.

When you'll look on the full comment before the external_url line, you'll see a link to the documentation:

## GitLab URL
##! URL on which GitLab will be reachable.
##! For more details on configuring external_url see:
##! https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab
external_url 'http://gitlab.example.com'

In the default gitlab.rb file (just after installing the Omnibus GitLab package) and also in the linked documentation you can see, that the external_url value is assigned without the = sign. It's done like that because the external_url is a method, not a variable. So here you're not assigning the value, but calling the method with this value. Ruby allows to call a method without brackets around the arguments, and this is a commonly used pattern when designing a DSLs based on Ruby.

When you try to set the value with:

external_url = 'https://server:actual-fqdn-of-gitlab-server-where-git-repo-is'

then the external_url method is not receiving any argument and in that case Omnibus GitLab falls back to a default value which is based on host's hostname.

In most cases the hostname of the server where GitLab is installed will be the same as the hostname that the server is accessible with. In some cases - including yours - the external_url needs to be set explicitly.

So the solution seems to be:

Change external_url = 'URL_HERE' to external_url 'URL_HERE'
Execute sudo gitlab-ctl reconfigure

This should set the proper values in gitlab.yml file and resolve your problems.


What role do Configuration Managment tools play in immutable infrastructure?

There are a few answers to that:

Something needs to build those immutable images. It is certainly easier to use old-school-style procedural scripting to build something when starting from a known starting state but this can still get very unwieldy over time (e.g. Dockerfiles), especially when you end up wanting a big matrix of different images for things like different versions of software, different environments, etc. Packer and other image build tools integrate nicely with Chef, Ansible, Puppet, Salt, and more.
Immutability is a spectrum, not a binary. Even in a "very immutable" deploy it is not uncommon to have some configuration files that need runtime management. This is also a place where you can use CAPS tools, though lighter-weight options like Consul Templates or etcd might make more sense depending on the overall infra. If you're doing immutable app servers but your database servers are more traditionally managed using Chef, then it might make sense to use Chef even on the immutable side for minor management tasks.
Zero-day management. Immutability is nice, but what are you going to do when the next OpenSSL 0day drops? If you have your build pipeline in a position to create immediate hotfix images and deploy them, then that's great. But many people will probably not have that kind of quick-turn ability.
Things that can't be immutable. The un-answer, but a whole infra is rarely 100% immutable. Things like database servers and developer workstations (yes, those are part of your infra) are somewhere between difficult and impossible to make immutable.


Command returns 0.0.0.0 for IP but only for leader

I ended up using {{.ManagerStatus.Addr}} and stripping the port number from the results with sed.

for NODE in $(docker  node ls -f 'role=manager' --format '{{.Hostname}}'); do
  docker node inspect --format '{{.ManagerStatus.Addr}}' ${NODE} | sed 's/:2377//'
done


Terraform foreach multiple map variables

Is something like the below what you are looking for?
locals {
  apps = keys(var.app_config)
}

resource "azurerm_key_vault_secret" "app_id" {
  for_each = toset(local.apps)

  name         = each.key
  value        = var.keyvault_secrets[var.environment][each.key]["application_id"]
  key_vault_id = data.azurerm_key_vault.mykv.id
}


One Repository for Kubernetes configuration or one configuration per application repository

If you're just migrating I'd also recommend using helm.
When using helm create the chart definition within the source code repository of the application, https://helm.sh/docs/helm/helm_create/.
With an increasing number of services it can become harder and harder to manage so people have started to embrace a "monorepo" that contains a definition of the all the helm charts they currently have deployed and their respective versions. This way you can easily track when a version of service A was bumped and when a version of service B was bumped. https://www.infracloud.io/blogs/monorepo-ci-cd-helm-kubernetes/


Include pre- and post_tasks transparently for all Ansible roles

Not tested this, but is include_role any help? so instead of a playbook like this,
 ---
 - name: Example
   hosts: all
   roles:
    - first_role
    - second_role
    - third_role

maybe this,
 ---
 - name: Example
   hosts: all
   tasks:
    - etckeeper_task
    - include_role:
        name: first_role
    - etckeeper_task
    - include_role:
        name: second_role
    - etckeeper_task
    - include_role:
        name: third_role
    - etckeeper_task

alternatively (also untested) you could build a role, let's call it wrap_role with tasks/main.yml containing,
---
# Replace this task with your actual before-role etckeeper task,
- name: etckeeper pre-actions
  meta: noop

- include_role:
  name: {{role_name}}

# Replace this task with your actual after-role etckeeper task,
- name: etckeeper post-actions
  meta: noop

Now your top-level playbook can be something like,
 ---
 - name: Example
   hosts: all
   roles:
     - { role: wrap_role, role_name: first_role }
     - { role: wrap_role, role_name: second_role }
     - { role: wrap_role, role_name: third_role }

not ideal based on your question, but at least it leaves the roles untouched, even if you have to alter top-level playbooks. If I get the time I'll test the above suggestions and update this answer accordingly.


Was AWS Secrets Manager removed from AWS CLI commands?

AWS secrets are still accessible via the CLI.

The link you provided goes to an old version of the AWS CLI.  Check out the latest version and commands here: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/secretsmanager/index.html


What is the impact of using Helm Deployments instead of StatefulSets for Databases like MongoDB or MySQL?

Back when Kubernetes announced the new StatefulSet feature with K8s v1.5 (converting it from the old PetSet name), they put out a really good blog post walking through an example of its usage: https://kubernetes.io/blog/2016/12/statefulset-run-scale-stateful-applications-in-kubernetes/

The first paragraph has a really good description of differentiating features that always stuck out in my mind:

  ...“at most one pod per index” for deployment of the Pods in the set. Along with ordered deployment, ordered termination, unique network names, and persistent stable storage ...

Break down the important ones:

Unique network names

In a deployment, it doesn't matter the network name as any pod should be able to pick up the work. However, when you are running things like a stateful cluster, you might have the concept of 'masters' and 'slaves'. This unique network naming will allow you to keep the distinction by referencing a specific pod, with a predictable incremental name.

Ordered Termination

This is important for re-deployments in a stateful cluster as you could have critical machines that need to stay up longer. If you have coordinator or 'leader' nodes, those should be the last to be restarted. StatefulSets allows you to do this. The linked blog post even shows an example of this.

Persistent stable storage

Arguably, this is the most crucial difference. By default, StatefulSets retain their storage. Kubernetes does not automatically delete the storage that was attached to a StatefulSet. The same can be achieved with deployments, but its not the default.


Expose metrics for Kotlin using JMX_Exporter to prometheus

After trying many things, I came to know that I needed to expose the port of my application's container in order to let Prometheus or other deployments to know. After exposing the port, I could see my application under targets on Prometheus and I could scrape all JMX and JVM metrics. Hope this would help someone in future...


How to override default aws region credentials for Terraform?

I believe you would be using multiple modules to create various resources(EC2,ELB, Etc) in AWS using Terraform. You can create a custom module having output value of each variable you want to configure per your requirement like aws-region, etc.

Then source(import) this custom module into your resource module and use its values using $.

custom_module

output "aws_region" {
  value = "us-xyz-1"
}
output "aws_anyproperty" {
  value = "abc"
}

Add above custom_module as source in your resource module

module "custom_module" {
  source="../../../modules/custom_module"
}

provider "aws" {
  version="~> y.x"
  profile="${module.custom_module.aws_anyproperty}"
  region="${module.custom_module.aws_region}"
}


Provision and configure Azure VM in private subnet (file, remote-exec, cloud-init, etc.)

Hashicorp recommends using provisioners only as a last resort. They recommend using the Azure VM custom_data method, or cloud-init where available.
Ansible would likely be overkill unless you will need ongoing in-place configuration management. But there are ways to make VM2 securely accessible to Ansible or provsioners if necessary.


My personal opinion is that each deliverable should reside in its own repository. However if one would like to keep the monorepo one could use changeset in conjunction with when:

pipeline {
    agent any
    stages {
        stage('build matchengine') {
            when {
                changeset "**/matchengine/*.*"
            }
            steps {
                echo 'building match engine'
            }
        }
        stage('build posttrade') {
            when {
                changeset "**/posttrade/*.*"
            }
            steps {
                echo 'building post trade'
            }
        }
    }
}

or change the directory in several build stage and build the code inside this directory:

stage('Build') {
  dir('web_app') {
    sh 'ls'
    sh 'git pull'
  }
}

One could also decide to use the parallel option to ensure that the build times will be shortened:

/* .. snip .. */
stage('run-parallel-branches') {
  steps {
    parallel(
      a: {
        echo "This is branch a"
      },
      b: {
        echo "This is branch b"
      }
    )
  }
}
/* .. snip .. */

Regarding the question:

  Set up Jenkins job which will trigger task only for repo where PUSH
  has occurred. e.g if my monorepo consist 2 merged repos repo1 and
  repo2 , so if any changes in repo1 should trigger only related task
  for repo1 and likewise for repo2.

I think that changeset would be most useful: a change will trigger the build, but as a change only occurred in a certain path, only that build will be started.


How to upload existing local project to a remote existing and empty github repository

When you create a new repository in Github, all the steps required are shown:

You just need to modify this accordingly for your use case:

you already have files, so no creating files
you may want to create the correct .gitignore for your project type, usually you exclude temporary files, such as caches, logs as well as files containing credentials
For this to work as per SSH example, you already need your primary public key uploaded to the same github account. If not, you can do this with a Github token, or look at the https instructions
if the existing repository already has commits in it, are you supposed to wipe them out? In this case use -f on git push.


Can I keep a container started with --rm option?

No, docker will delete the container when it stops, and there's no update option to change that behavior. You can see this in the help output:

$ docker container update --help

Usage:  docker container update [OPTIONS] CONTAINER [CONTAINER...]

Update configuration of one or more containers

Options:
      --blkio-weight uint16        Block IO (relative weight), between 10 and 1000, or 0 to disable (default 0)
      --cpu-period int             Limit CPU CFS (Completely Fair Scheduler) period
      --cpu-quota int              Limit CPU CFS (Completely Fair Scheduler) quota
      --cpu-rt-period int          Limit the CPU real-time period in microseconds
      --cpu-rt-runtime int         Limit the CPU real-time runtime in microseconds
  -c, --cpu-shares int             CPU shares (relative weight)
      --cpus decimal               Number of CPUs
      --cpuset-cpus string         CPUs in which to allow execution (0-3, 0,1)
      --cpuset-mems string         MEMs in which to allow execution (0-3, 0,1)
      --kernel-memory bytes        Kernel memory limit
  -m, --memory bytes               Memory limit
      --memory-reservation bytes   Memory soft limit
      --memory-swap bytes          Swap limit equal to memory plus swap: '-1' to enable unlimited swap
      --pids-limit int             Tune container pids limit (set -1 for unlimited)
      --restart string             Restart policy to apply when a container exits

Trying to change the restart policy will result in an error:

$ docker run --rm --name test-rm -d busybox tail -f /dev/null
14a7d9bfebc3356652e3c3f060e92d9cebb9f3eb4490873540c9eb1c272f6612

$ docker update --restart=unless-stopped test-rm
Error response from daemon: Cannot update container 14a7d9bfebc3356652e3c3f060e92d9cebb9f3eb4490873540c9eb1c272f6612: Restart policy cannot be updated because AutoRemove is enabled for the container

You can try to copy your data out of the container with:

docker container cp $container_name:/path/in/container /path/on/host

If you don't know which files to save, list which files have changed in the container with:

docker container diff $container_name


Where is it said I can safely the same local volume to multiple docker containers?

By a "local volume" I assume you mean a block device directly attached to the host, such as a "normal" internal disk.

The standard way to run multiple Docker, containers now is Kubernetes, since the demise of Docker Swarm. My reference here is the Kubernetes documentation for Persistent Volumes specifically the table on Access Modes, and the column for ReadWriteMany. A local volume would be a HostPath which is not supported for this use case. However if you were using iSCSI it would support ReadOnlyMany, and if you used NFS you could get ReadWriteMany.


Trigger Jenkins job from Gitlab CI with parameters

Here's the way I do it: no plugin required, just triggering Jenkins api from gitlab-ci.

Gitlab-CI

I will assume you have a gitlab-ci runner installed and configured.

First, you need to have a .gitlab-ci.yml file in your project having a basic structure such as:

stages:
- my-jenkins-trigger
variables:
 MY_VARIABLE: "EVERYTHING_IS_AWESOME"

my-jenkins-trigger-job:
 stage: my-jenkins-trigger
 script: curl -i -X POST --user JENKINS_USER:JENKINS_TOKEN JENKINS_JOB_URL/buildWithParameters?MY_JENK_PARAM=${MY_VARIABLE}

In the above, I also assume

You have a Jenkins job somewhere at URL JENKINS_JOB_URL
this Jenkins job has a build parameter called MY_VARIABLE
JENKINS_USER, JENKINS_TOKEN are defined [*]

That simple?

Well yes, but no...

That is the rough structure. That script will merely trigger a Jenkins job and forget about it. You need to work a little more to monitor the job and feed its status back in Gitlab-CI, manage security and possibly get some commit info from gitlab to inject into your job.

Monitoring

In order to have a proper monitoring, I recommand to write a full trigger + monitor + return value script [** ] (in whatever language available or you're familiar with).

Just start by triggering the job as I stated above.

Then, run a while loop (don't forget to put it to sleep [***]) on

curl --silent --user JENKINS_USER:JENKINS_TOKEN JENKINS_JOB_URL/lastBuild/api/json | grep result\":null > /dev/null

until the result of this command is not 0.

Once the Jenkins job is finished, you would probably want to fetch the job's console in Gitlab

curl -i -X POST --user JENKINS_USER:JENKINS_TOKEN JENKINS_JOB_URL/lastBuild/consoleText

Finally you may curl once more on JENKINS_JOB_URL/lastBuild/api/json but this time you grep it on UNSTABLE, SUCCESS or FAILURE.

Discussion

By following the guidelines above, you can fully orchestrate Jenkins jobs from Gitlab-CI. I've posted a long discussion on why and when should you do this.

I hope this will help you.

[*] Your Gitlab project Settings > CI/CD > Secret vaiables
[** ] Of course I mean by that to craft a script nicely with params, functions, nice variable names, meaningful logs... You name it.
[***] I found a sleep of 20 seconds worked for me


Yes, one would usually have minimal docker images, i.e., one responsibility per image/container, with appropriate networking between them.
Yes, images are linear, there is only ever one unbroken line of FROMs. To merge two images, get the two Dockerfiles and see what they are doing; it should be pretty self explanatory. Then you can either craft a single new Dockerfile mixing and merging lines from the sources, or keep one of the original dependencies as your FROM, and just copy lines from the other original to a new Dockerfile.
You would use command in a docker-compose.yml in the same cases when you would give a command on the docker run line, in case you were running the container manually. That is, if you have an image which conveniently supports different commands, and were it would just be a waste to have a different image per command. So... if you find that you run your container with the same command in 99% of all cases, then put that CMD into the Dockerfile and remove it from the docker-compose.yml. If you find, instead, that you are using different commands all the time, then keep them in docker-compose.yml.